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(57) Altsiruct: Arrangement for obtaining information about objects (200) in an environment around a vehicle (100), e.g., in blind 
sjuiis of ihc driver of the vehicle, including one or more light emitting components (I KM )4) arranged on the vehicle for emitting 
infrared light (117. I IS) into the environment around the vehicle and receivers (I KM 14) arranged on the vehicle for receiving 
infrared light from the environment around the vehicle. The information about the objects is obtained based on analysis of the 
received in I'm red lighi by a processor ( 120), e.g.. the distance between the vehicle and the object, the velocity of the object and the 
idenlily of the object. Pattern recognition techniques are preferably used to obtain the desired information. Control of the vehicle 
( 100) can be effected based on the detection of particular objects and their position and velocity, e.g.. an audio or visual warning 
device or a steering wheel control device (146). 
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VEHICULAR BLIND SPOT IDENTIFICATION AND MONITORING SYSTEM 

FIELD OF THE INVENTION 

This invention relates to collision avoidance systems and specifically to the detection of objects in 
5 the various blind spots that surround a truck or automobile and the warning or prevention of the vehicle 
operator from taking an action such as changing lanes when such an action might lead to an accident. 

BACKGROUND OF THE INVENTION 

During the process of operating a motor vehicle, it is necessary for the operator to obtain 
10 information concerning the proximity of various dangerous objects and their relative velocities for the 
operator to may sound driving decisions, such as whether or not there is enough time to change lanes. This 
information should be obtained from the area that completely surrounds the vehicle. In order to gather this 
information, the operator is frequently required to physically turn his or her head to check for occupancy of 
a blind spot, for example. In taking such an action, the attention of the driver is invariably momentarily 
15 diverted from control of the vehicle. 

For an automobile, the blind spots typically occur on either side of the vehicle starting 
approximately at the position of the driver and extending backwards sometimes beyond the rear of the 
vehicle. These blind spots depend heavily on the adjustment of the angle of the mirror. Different areas are 
in the blind spot depending on the mirror angle. Since it is in general not known whether or how the 
20 mirror is set for the particular vehicle, a blind spot detector must detect objects anywhere along the side of 
the vehicle regardless of the mirror setting. 

The problem is more complicated for trucks that not only have much larger blind spots along the 
sides of the vehicle but also have a serious blind spot starting in front of the right front bumper of the 
vehicle and extending beyond the right door. This blind spot is particularly serious in urban driving where 
25 small vehicles, motorcycles, pedestrians, bicycles etc. in this area can be completely hidden from the view 
of the driver. 

Many systems have been designed which attempt to rotate the mirror to pick up or allow a driver 
to visually see the object in the blind spot. This is difficult to do without knowledge of the location of the 
eyes of the driver. For most systems that do not incorporate an occupant sensor capable of determining the 
30 location of the driver's eyes, there is a risk that the mirrors will be positioned wrongly thus exacerbating 
rather than helping the blind spot detection problem. Also, a system that rotates the mirror will make the 
driver nervous since he or she will not be able to see the scene that he or she is accustomed to seeing in the 
mirror. 

Many systems have also been proposed that display a view of the blind spot, using a video 
camera, onto a display either on the instrument panel or on the windshield as a "heads-rip" display. Any 
system which displays a picture of the object on the screen that is inside the vehicle is also going to 
confuse the driver since he or she will not be able to relate that picture to an object such as another vehicle 
in the blind spot on the side of the host vehicle. Additionally, the state of the art of such displays does not 
provide equally observable displays at night or in bright sunlight. Thus, displays on a CRT or LCD are not 



35 



1 



WO 01/85491 



« i 



PCT/US01/14876 



natural and it is difficult for a driver to adjust to these views. The lighting of the views is too faint when 
sunlight is present and too bright when the vehicle is operating at night. Therefore, none of these 
television-like displays can replace the actual visual view of the occupant. 

Other systems that are based on radar or ultrasound have also not been widely adopted for reasons 
5 related to cost, accuracy and false alarms. Both systems use beams of energy that can become several feet 
in diameter by the time that they reach the edges of the blind spot and thus can confuse a large vehicle or a 
guardrail, sign, parked car etc. two lanes over with a vehicle in the blind spot. Some such systems attempt 
to filter threatening objects from non-threaternng objects based on the relative speed of the object and thus 
err by eliminating a significant number of such threats. A tradeoff exists in all such systems where, if all 
1 0 threatening objects are made known to the driver, the false alarm rate becomes unacceptable and the driver 
soon loses confidence in the system and ignores it. If the false alarm rate is kept low, many dangerous 
situations are ignored. 

Thus, all the prior art systems discussed above have serious failure modes. The lesson is that if a 
vision-based system such as the rear view mirror is going to be replaced with a non-vision system then the 
15 non-vision system must be almost as good as the vision system. 

Some other problems come about when a vehicle strays into the lane of the host vehicle. Most 
systems will fail to warn the operator and thus an accident can result. Thus, the blind spot problem is 
really two problems relating to the motion of the potentially striking vehicle and the potentially struck 
vehicle. 

20 The problem that is addressed by this invention is to determine what information is needed about 

the object that is in the blind spot and then the manner in which this information is presented to the vehicle 
operator so as to eliminate accidents caused by the failure of the operator to see such an object. This 
information includes the accurate location of the object relative to the host vehicle, its size, its relative 
and/or absolute speed, and the identity or kind of object. This information must be known regardless of 

25 the changes in road geometry such as steep hills and sharp curves or changes in environmental conditions. 
Naturally, the system must be low cost if it is going to be purchased by the public. 

Studies have shown that giving the driver an extra half-second could eliminate as many as 50 
percent of the accidents. Thus, the risk of an accident must also be communicated to the operator in a 
timely fashion to permit the driver to take evasive action or not take a particular action such as a lane 

30 change. 

What is needed therefore is a system that acts like the eyes of the driver and interprets the 
situation and only gives a warning when there is a real possibility of an accident. A passive warning can 
be given in the form of a light on the mirror whenever an object is in the blind spot but an active signal 
such as an audible signal or an intervention in the steering of the automobile can only be provided when it 
35 is necessary to prevent the accident. This system must work with very high reliability and accuracy since 
the consequences of an error can be serious injuries or death. 
1 . Prior Art on Blind Spot Detection Systems 

In "A Survey of Automotive Collision Avoidance Systems" by Uttamkumar Dravidam and Sabri 
Tosunoglu of the Florida International University, the authors provide a good review of the field of 
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obstacle sensors. What follows is a summary of their analysis. Obstacle sensors such as used for blind 
spot detection can be divided into three types which will now be described: 

Optical sensors include passive infrared, laser radar and vision. They generally are sensitive to 
external environmental conditions, which may not be a problem for blind spot detection since the objects 
to be detected are usually nearby the host vehicle. Passive infrared and vision cannot provide a direct 
measurement of distance to an object unless part of the field of view is muminated by a point or structured 
light Laser radar does provide the capability of direct distance measurement, as will be described below, 
and a stereo camera can also provide distance information. 

AMCW (amplitude modulated continuous wave), FMCW (frequency modulated continuous 
wave) and impulse radar are not generally affected -by adverse environmental conditions. Although 
relatively expensive, FMCW radar is a good technique for long-range distance measurement providing the 
object to be measured can be sorted out. Radar in general has a high false alarm rate due to the large pixel 
size at any significant distance from the host vehicle, to multipath effects and reflections off of signs, 
bridges, guardrails etc. 

15 Ultrasonics is good in applications where only short relative distance measurements are required, 

because it is able to provide high distance to the target resolution for a relatively low cost. However, for 
imaging applications, the slow speed and relatively large pixel size renders ultrasonics marginal. Also, 
ultrasonic waves can be significantly distorted by thermal gradients and wind. 

Various researchers have attempted combinations of these technologies with the particular 

20 combination of laser radar and pulse or FMCW being quite advantageous for long distance collision 
avoidance applications. 

What follows in a brief description of the principles of operation for different types of sensors 
including their main advantages and disadvantages. For blind spot applications, sensors should be able to 
accurately determine the location of the object and the speed of the obstacle relative to the host vehicle. 
25 How well this is achieved can be measured with the following indicators: 

Sensing range: the maximum and minimum range over which the technique can be used. 

Range Resolution: the relative change in range that can be measured. 

Pixel Resolution: the width of the beam or size of the pixel received and to which the sensor is 
sensitive. 

30 Response time: how quickly the sensor can respond to a change in the blind spot occupancy. 

Ultrasonics: These sensors work by measuring the time-to-flight of a short burst of ultrasound 
energy typically at a frequency of 40-200 kHz. The time taken for the ultrasonic waves to travel to and 
return from the obstacle is directly, proportional to the distance between the obstacle and the host vehicle. 
The main advantage is their relative low cost and small size. These sensors are also very sensitive to 
35 changes in the density of air that can be caused by, e.g .. high wind velocity and temperature gradients. 

Passive Infl-ared: These sensors measure the thermal energy emitted by objects. Their main 
advantage is their low cost and small size, and main disadvantage is their inability to determine the 
distance to a detected object and slow response time. 

Laser Radar: As with regular radar, two techniques exist: (1) a pulsed-beam of infrared lioht 
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coupled with time-of-flight measurements, and (2) the modulation of a continuous light beam. The pulsed 
technique offers long range, high directionality, and fast response time. Its limitations are its sensitivity to 
environmental conditions. 

FMCW or AMCW Radar'. This type of radar uses modulated microwave frequencies, so that the 
5 frequency difference between the reflected and the transmitted signal is proportional to the relative velocity 
of the object. When two waves of slightly different frequencies are used, the distance to the object can 
also be determined by the phase relationship between the two received reflections. Despite its high cost, 
this technique offers the advantages of being insensitive to environmental conditions, but the disadvantage 
of having a large pixel size. 

1 0 Impulse Radar. This radar differs from FMCW in that it uses very short pulses instead of a 

continuous wave. Like FMCW radar, it is insensitive to environmental conditions, but the cost is 
significantly lower than FMCW. Distance can be determined by time-of-flight measurements and velocity 
can be determined from successive distance measurements. It also has the disadvantage of having a large 
pixel size resulting in a high false alarm rate and too Utile information to permit object identification. 

15 Capacitive and Magnetic: Capacitive and magnetic sensors are able to detect close objects (within 

about 2 m.), using the capacitance or magnetic field variations between electrodes excited at low 
frequencies, typically about 5 kHz. Despite their limited range, they are low in cost, and robust to external 
environmental effects. The poor resolution compared to other techniques makes it unlikely that these 
devices will be used for blind spot detection since most objects are close to the vehicle. 

20 Vision Systems: These techniques are based on the use of a camera and image-processing 

software. They are sensitive to external environmental conditions; however, this is not a significant 
shortcoming for blind spot detection. 

Prior art related to the invention includes: U.S. Pat. Nos. 4,766,421; 4,926,170; 5,008,678; 
5,087,918; 5,122,796; 5,229,975; 5,235,316; 5,289,321; 5,291,261; 5,311,012; 5,325,096; 5,122,796; 

25 5,354,983; 5,418,359; 5,424,952; 5,463,384; 5,467,072; 5,517,196; 5,668,539; 5,670,935; 5,675,326; 
5,765,116 and 6,038,496 and International Publication Nos. WO 90/13103. WO 95/25322 and WO 

• * 

99/42856. 

None of the prior art described in tins references discloses a method or apparatus of monitoring 
the blind spot of the vehicle that analyzes an image of one or more objects that occupy the blind spot, 
30 identifying them and determining the location and relative velocity of the objects relative to the host 
vehicle in a manner that permits an accurate warning to be issued to the driver of a potentially dangerous 
situation. 

2. Definitions 

Preferred embodiments of the invention are described below and unless specifically noted, it is 
35 the applicants 5 intention that the words and phrases in the specification and claims be given the ordinary 
and accustomed meaning to those of ordinary skill in the applicable art(s). If the applicants intend any 
other meaning, they will specifically stale they are applying a special meaning to a word or phrase. 

A "blind spot", for the purposes of this invention, will include those areas surrounding a vehicle 
that could contain an object that may not easily be seen by the driver through the various rear view mirrors 
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but which could pose a threat either to the vehicle occupants or to the occupants of the object, or other 
others such as pedestrians, in the blind spot 

Pattern recognition is commonly used in practicing the instant invention. "Pattern recognition" as 
used herein will generally mean any system, which processes a signal that is generated by an object, or is 
modified by interacting with an object, in order to determine which one of a set of classes that the object 
belongs to. Such a system might determine only that the object is or is not a member of one specified 
class, or it might attempt to assign the object to one of a larger set of specified classes., or find that it is not 
a member of any of the classes in the set. The signals processed are generally electrical signals corning 
from transducers which are sensitive to either acoustic or electromagnetic radiation and, if electromagnetic, 
they can be either visible light, infrared, ultraviolet or radar or low frequency radiation as used in 
capacitive or magnetic based sensing systems. 

A trainable or a trained pattern recognition system as used herein means a pattern recognition 
system which is taught various patterns by subjecting the system to a variety of examples. The most 
successful such system is the neural network. Not all pattern recognition systems are trained systems and 
15 not all trained systems are neural networks. Other pattern recognition systems are based on fuzzy logic, 
Kalman filters, sensor fusion, correlation as well as linear and non-linear regression. Still other pattern 
recognition systems are hybrids of more than one system such as neural-fuzzy systems. 

To "identify" as used herein will mean to determine that the object belongs to a particular set or 
class. The class may be one containing, for example, all human objects, automobiles, trucks, bicycles, 
20 motorcycles, guard rails, trees, etc. depending on the purpose of the system. In the case where a particular 
vehicle such as a fire engine is to be recognized, the set or class will contain only a single element, i.e., the 
vehicle to be recognized. 

An "occupying item" of a blind spot may be an automobile, truck, motorcycle, pedestrian, 
bicycle, animal, guard rail, tree, utility pole, as well as many other objects. 

25 In ^ description herein the term "approaching" when used in connection with the mention of an 

object or vehicle approaching another will mean the relative motion of the object toward the vehicle having 
the sensor system. In other words, the coordinate system used in general will be a coordinate system 
residing in the vehicle. This convention permits a general description to cover all of the cases such as 
where a moving vehicle approaches a stationary vehicle or where both vehicles are moving. 

30 An "electronic shutter" or "light valve" as used herein will mean any method of controlling the 

amount of light that can pass through the device based on an electronic signal control of the device. 
3. Pattern recognition prior art 

. Neural networks as the pattern recognition technology are used in several of the implementations 
of this invention since it makes the monitoring system robust, reliable and practical. The resulting 
35 algorithm created by the neural network program- is usually only a few hundred lines of code written in the 
C or C++ computer language. The resulting systems are easy to implement at a low cost, making them 
practical for automotive applications. The cost of the CCD and CMOS arrays, for example, have been 
expensive until recently, rendering their use for blind spot systems impractical. Similarly, the 
implementation of the techniques of the above referenced patents frequently requires expensive 
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microprocessors while the implementation with neural networks and similar trainable pattern recognition 
technologies permits the use of low cost microprocessors typically costing less than $10 in large quantities. 

The present invention preferably uses sophisticated trainable pattern recognition capabilities such 
as neural networks. Usually the data is preprocessed, as discussed below, using various feature extraction 
5 techniques. An example of such a pattern recognition system using neural networks on sonar signals is 
discussed in two papers by Gorman, R. P. and Sejnowski, T. J. "Analysis of Hidden Units in a Layered 
Network Trained to Classify Sonar Targets", NeuraL Networks, Vol.1, pp 75-89, 1988, and 'Teamed 
Classification of Sonar Targets Using a Massively Parallel Network", IEEE Transactions on Acoustics, 
Speech, and Signal Processing, Vol. 36, No. 7, July 1988. Examples of feature extraction techniques can 
10 be found in U.S. Pat. No. 4,906,940 to Green et al. Examples of other more advanced and efficient pattern 
recognition techniques can be found in U.S. Pat. No. 5390,136 and U.S. Pat. No. 5,517,667 to S.T. Wang. 
Other examples include U.S. Pat. Nos. 5,235,339, 5,214,744, 5,181,254, and 4,881,270. 
4. Optics 

Optics can be used in several configurations for monitoring the exterior of an automobile. The 
15 receiver can be a CCD or CMOS imager, to receive the emitted or reflected light. A laser can either be 
used in a scanning mode, or, through the use of a lens, a cone of light can be created which covers a large 
portion of the object in the blind spot. In these configurations, the light can be accurately controlled to 
only illuminate particular positions of interest on the vehicle. In the scanning mode, the receiver need only 
comprise a single or a few active elements while in the case of the cone of light, an array of active 
20 elements is needed. The laser system has one additional significant advantage in that the distance to the 
illuminated object can be determined as disclosed in U.S. Pat. No. 5,653,462. 

In a simpler case, light generated by a non-coherent light emitting diode (LED) device is used to 
illuminate a desired area. In this case, the area covered is not as accurately controlled and a larger CCD or 
CMOS array is required. Recently, the cost of CCD and CMOS arrays has dropped substantially with the 
25 result that this configuration is now the most cost-effective system for monitoring the blind spot as long as 
the distance from the transmitter to the objects is not needed. If this distance is required, then a laser 
system using modulation and phase detection or time-of-flight techniques, a stereographic system, a 
focusing system, a combined ultrasonic and optic system, or a multiple CCD or CMOS array system as 
described herein is required. 

30 In a particular implementation, the illuminating light is in the form of a modulated infrared laser 

light that is scanned in a line that illuminates an object in the blind spot. The reflected light is received by 
a pin diode after passing through a narrow frequency band notch filter. The diode is a single pixel device 
but since the direction of the transmitted light is known, the direction of the reflected light is also known. 
The phase of received light is then compared with the transmitted light. The modulating frequency was 

35 selected so that no more than one wavelength of light exists within the blind spot area. The location of the 
reflecting object can then be determined by the phase difference between the transmitted and reflected 
light. Although the described system uses a linear scan, it is also possible to use a two-dimensional scan 
and thereby obtain a three-dimensional map of the blind spot. This can be done using a pin diode as 
described or the light received by the CMOS array can be monitored on a pixel by pixel basis in a manner 
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similar to the PMD system described in Schwarte, et al. "New Powerful Sensory Tool in Automotive 
Safety Systems Based on PMD-Technology". In this latter case, the entire blind spot area may be flooded 
with modulated infrared light as described in the paper. On the other hand, it is difficult to overcome the 
light from natural sources such as the sun by a single flood light source and therefore a line or even a 
5 scanning point source permits better distance measurement using a light source of reasonable intensity. 

A mechanical focusing system, such as used on some camera systems can determine the initial 
position of an object in the blind spot A distance measuring system based of focusing is described in U.S. 
Pat. No. 5,193,124 (Subbarao) which can either be used with a mechanical focusing system or with two 
cameras. Although the Subbarao patent provides a good discussion of the camera focusing art it is a more 
10 complicated system than is needed for the practicing the instant invention. In fact, a neural network or 
optical correlation system can also be used to perform the distance determination based on the two images 
taken with different camera settings or from two adjacent CCD's and lens having different properties as 
the cameras disclosed in Subbarao making this technique practical for the purposes of this instant 
invention. Distance can also be determined by the system described in U.S. Pat. No. 5,003,166 by the 

1 5 spreading or defocusing of a pattern of structured light projected onto the object of interest. Distance can 
also be measured by using time-of-flight measurements of the electromagnetic waves or by multiple CCD 
or CMOS arrays as is a principle teaching of this invention. 

In each of these cases, regardless of the distance measurement system used, a trained pattern 
recognition system, as defined above, can be used to identify and classify, and in some cases to locate, the 

20 illuminated object. 

5. Optics and acoustics 

Both laser and non-laser optical systems in general are good at determining the location of objects 
within the two-dimensional plane of the image and a pulsed laser or continuous modulated radar system 
mthe scanning mode can determine the distance of each part of the image from the receiver by measuring 

25 the time-of-flight or by phase measurement. It is also possible to determine distance with the non-laser 
system by focusing as discussed above, or stereographically if two spaced apart receivers are .used and, in 
some cases the mere location in the field of view can be used to estimate the position of the object in the 
blind spot, for example. Finally, a recently developed pulsed quantum well diode laser also provides 
inexpensive distance measurements. 

30 Acoustic systems are additionally quite effective at distance measurements since the relatively 

low speed of sound permits simple electronic circuits to be designed and minimal microprocessor 
capability is required. If a coordinate system is used where the z-axis is from the transducer to the object, 
acoustics are good at measuring z dimensions while simple optical systems using a single CCD are good at 
measuring x and y dimensions. The combination of acoustics and optics, therefore, permits all three 
measurements to be made from one location with low cost components as discussed in commonly assigned 
U.S. Pat. Nos. 5,S45 ? 000 and 5,835,613. 

One example of a system using these ideas is an optical system that uses natural light coupled 
with a lens and CCD or CMOS array which receives and displays the image and an analog to digital 
convener (ADC), or frame grabber, which digitizes the output of the CCD or CMOS and feeds it to an 
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artificial neural network (ANN) or other pattern recognition system for analysis. This system uses an 
ultrasonic transmitter and receiver for measuring the distances to the objects located in the blind spot. The 
receiving transducer feeds its data into an ADC and from there, the converted data is directed into the 
ANN. The same ANN can be used for both systems thereby providing full three-dimensional data for the 
5 ANN to analyze. This system, using low cost components, will permit accurate identification and distance 
measurements not possible by either system acting alone. If a phased array system is added to the acoustic 
part of the system, the optical part can determine the location of the object and the phased array can direct 
a narrow beam to the location and determine the distance to the object through time-of-flight. 

Although the use of ultrasound for distance measurement has many advantages, it also has some 

10 drawbacks. First, the speed of sound limits the rate at which the position of the object can be updated. 
Second, ultrasound waves are diffracted by changes in air density that can occur when thermal gradients 
are present or when there is a high-speed flow of air past the transducer. Third, the resolution of 
ultrasound is limited by its wavelength and by the transducers, which are high Q tuned devices. Typically, 
the resolution of ultrasound is on the order of about 2 to 3 inches. Finally, the fields from ultrasonic 

15 transducers are difficult to control so that reflections from unwanted objects or surfaces add noise to the 
data. 

* 

6. Discussion of the Blind Spot problem and Solutions 

The above review of a prior art blind spot detecting systems illustrates that no existing system is 
believed to be sufficiently adequate. A fundamental problem is that vehicle operators are familiar with 

20 visual systems and inherently distrust all other technology. As soon as the non-visual system gives a false 
alarm or fails to detect an object in the blind spot the operator will cease to depend on the system. 
Theoretically, the best systems would be based on cameras that allow the operator to view all of the blind 
spots. However, there are no adequate display systems that will appear to the operator to be equivalent to 
an actual view of the scene. CRTs and LCDs require driver concentration and do not have the dynamic 

25 range of lighting that is comparable to the real world. Either the display will be too bright at night or too 
dim during daylight Although radar systems can accurately measure distance to an object, they are poor at 
placing the object in the lateral and vertical coordinates relative to the vehicle and thus create many false 
alarms. 

The simplest system must be able to accurately position the object in the blind spot relative to the 
30 host vehicle and inform the driver that a collision potential exists if the driver decides to change lines, for 
example. This warning must be given to the driver either at a place where he can almost subconsciously 
observe the warning when he is contemplating a lane change maneuver, or it must provide an audible 
warning if he attempts to make such a lane change maneuver. Finally such a system might even prevent a 
driver from executing such a maneuver. 
35 A more sophisticated system would be to provide a simple icon image of the host vehicle and all 

surrounding vehicles as viewed from above. In this manner, with a simple glance, the driver can determine 
the location and identity of all objects that are in his blind spot or in the vicinity of the vehicle in any 
direction. If this display is kept simple, then the problems of visual dynamic range become much less 
severe. That is, if the driver need only see dark objects on a white background and if the size of these 
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objects is significant, than the display could be viewed both at night and under daylight conditions. 

To accomplish these goals, it is necessary to positively locate an object in the blind spot and 
provide some identification as to what that object is. A driver will respond quite differently if the object is 
a guard rail or a line of parked cars then he will if it is a Porsche over taking him at 150 kph. 
5 Thus, the requirements of the system are to identify the object and to locate the object relative to 

the host vehicle. To identify the object preferably requires a pattern recognition system such as neural 
networks or optical correlation systems. To locate the object preferably requires some means of measuring 
the distance from the camera to the object. A CMOS camera is quite capable of rapturing an image of an 
object in the blind spot and if the camera is an HDRC camera, then it can perform well under all normal 
1 0 lighting conditions from midnight to bright sunshine especially if minimal iUumination is provided on dark 
nights. 

The measurement of the distance to the object can be accomplished many different ways 
including ultrasonically, using laser radar, FMCW or AMCW radar, and range gated pulsed micropower 
impulse radar. All of these distance measuring techniques as well as stereographic, focusing, structured 
15 light, triangulation and other similar techniques are envisioned for use in this invention 

A low-cost preferred approach of solving the distance measuring problem that is consistent with 
an HDRC camera system is to project onto the volume of the blind spot a series of infrared light pulses. 
These pulses are created by an array of laser diodes that are displaced from the camera in such a manner 
that a pulse of light reflected off of an object in the blind spot will appear on a certain pixel area in the 

20 camera field of view and since the location of the transmission of the pulse is known and the location of 
the camera is known, the distance to the reflecting surface is also known by triangulation. By a judicial 
choice of transmission angles from the laser diode array, the entire volume of the blind spot can be covered 
with sufficient accuracy so that no significant object can penetrate the blind spot without creating a 
reflection and thereby permitting the distance to the object to be determined. 

25 Although a series of pulses from a laser diode array are contemplated, other techniques will also 

accomplished the same goal, however, at a generally higher cost. For example, a continuous laser beam 
can be used that would scan the blind spot area m either one or two dimensions. Since the direction of the 
laser will be known that all times its reflection and excitation of pixels on the CMOS array would permit, 
once again, an accurate, mapping of the distance to various points on the object in the blind spot to be 

30 accomplished. This technique however requires a scanning laser system that in general, although more 
accurate, would be more expensive than a simple array of LEDs. Once again, the photonic mixing device 
described above would also provide a three-dimensional image of the contents of the blind spot as would a 
similar and preferred system described below. 

Another technique is to superimpose on the blind spot area a pattern of light commonly referred to 

35 as structured light. The source of the structured light must be displaced from the imaging array. By 
observing characteristics of the reflected pattern., such as the distances between portions of the pattern, the 
distance to the object can be determined. This system, although common in machine vision applications, 
requires greater computational resources then the simple LED array described above. Nevertheless., it is a 
viable approach and envisioned for use in the invention. 
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positively warn the driver that he or she is about to execute a maneuver that might result in an accident 
based on the fact that there is an object in his or her blind spot. 

A preferred implementation of the instant invention will use a passive optical system for 
monitoring the presence of objects in the blind spot Pattern recognition technologies such as neural 
5 networks and optical correlation systems will be used to positively identify the object that is in the blind 
spot. This object may be a pedestrian, bicyclist, motorcyclist, guardrail, animal, automobile, truck, or 
whatever. The system will be trained or otherwise programmed to inform the operator either optically or 
orally that such an object appears in the blind spot. It will also inform the driver as to which blind spot 
contains the object. The system can also inform the driver as to whether this object is moving or stationary 

10 in an absolute sense and/or also in relation to the host vehicle. This information can be presented to the 
operator in a variety of ways. Initially, a light or simple icon can appear on the rear view mirror indicating 
either that some object exists or that a particular object exists. In more sophisticated systems, an icon 
representing the object can be placed on a simple icon display which shows the vehicle from an overhead 
view and an icon which shows the blind spot object and its location. Alternately, an oral annunciation can 

15 be provided which tells the driver that, for example, there is a guardrail three feet to his left, or that there is 
an automobile approaching from the rear in an adjacent lane at a relative speed of 100 kph and is currently 
50 feet behind the vehicle. All of these types of warnings can be provided if an identification can be made 
of the object in the blind spot and an accurate measurement made of the position and velocity of that object 
relative to the host vehicle. This is an object of this invention. 

20 It can be seen from this description that the system of this invention will inform the driver of the 

type of object in the blind spot, where it is located specifically, what its velocity is relative to the host 
vehicle, and in more sophisticated systems, will show graphically an icon showing the object relative to the 
vehicle from an overhead view which is easily understandable by the driver with a mere glance at the 
display. Therefore, the system overcomes all of the objections and problems described above with respect 

25 to the prior art systems. 

The use of passive optical camera systems, such as the HDRC camera, has been discussed and the 
method of using either neural networks, optical correlation, or other pattern recognition systems has also 
been discussed that illustrates how, in the present invention, the identity of the object occupying the blind 
spot will be determined. What follows now is a more detailed discussion of how the position of that object 

30 will be determined. 

As discussed above, many technologies are available for measuring distance, some more 
expensive than others. The system described here is believed to be a low cost approach and a preferred 
embodiment of the invention. This invention is not limited to this technology, however, and any practical 
distance measuring system is contemplated. 

35 Although simple icon displays are contemplated by tins invention, this is due to die lack of 

. sophistication or capability of current display technology. In other words, the dynamic range of light that 
can be emitted by conventional displays is insufficient to display other than the most simple messages. 
Technology advances and it is expected that accurate color displays with high dynamic range will be 
available in the not too distant future. When such displays are available, a more accurate representation of 
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the object in the blind spot even to the point of an actual image might become feasible. 

This invention does hot generally contemplate the use of rear view mirrors to permit the vehicle 
operator to actually see the contents of the blind spot. This is because to accurately accomplish this 
requires knowledge of the position of the eyes of the driver. It is been observed that drivers adjust side 
5 rear view iriirrors over an extended range that renders the use of the mirror angle unsuitable for 
deterrnining the position of the driver's eyes. Furthermore, the driver may change position without 
changing the position of the rear view mirror. Occupant sensing systems are now being deployed on 
vehicles that have the capability of deterrnining the location of the eyes of a vehicle operator. For those 
vehicles that contain such a system, the possibility exists not only to automatically adjust the mirror to 

1 0 optimally display the contents of the blind spot, but also to change the orientation of the mirror when some 
object that the driver should be aware of is in the blind spot. This invention therefore contemplates such 
activities when occupant sensing systems are placed on vehicles. 

For the preferred implementation of the system, the light from laser diodes will actually cross the 
field of view of the camera. If there is a heavy fog, for example, in the vicinity of the blind spot, then the 

1 5 reflection of the light off of the fog will create an elliptical image on the camera sensor array. This would 
also be true when heavy rain or heavy snowfall is present. This fact can be used to determine visibility. 
Observations of visibility conditions of objects in the blind spot even during severe weather conditions has 
led the inventors of this invention to the conclusion that when the visibility is so poor that the passive 
optical system using laser diodes described herein is not functioning with sufficient accuracy, that the 

20 operator of the vehicle should not be operating the vehicle on the roads and therefore the vehicle operator 
should be informed that safe travel is not possible. Thus, the use of radar or other technologies to view the 
blind spot, which is actually quite close to. the vehicle, is not necessary since vehicle operation should not 
be permitted when the visibility is so poor that the object cannot be seen in the blind spot by the system of 
this invention. Nevertheless, this invention can contribute to safe driving in these conditions, if such 

25 driving is attempted, since an indication will be obtained by the system based on the elliptical reflections 
from the laser diode indicating that the visibility is unacceptable. 

There will be conditions when the optical system from the CMOS camera has deteriorated due to 
contamination on the lens. Similarly, the light emitting laser diodes will emit less light if the lenses are 
soiled. The system of this invention contemplates a continuous diagnostic feature that will permit sensing 

30 of either of these conditions. This can be accomplished in a variety of ways such as a laser diode aimed at 
the road surface close to the vehicle but within view of the CMOS camera. If the reflection over a period 
of time is not sufficient, then a warning light will appear on the instrument panel informing the driver that 
maintenance is required. Naturally, there are many other methods by which a similar diagnostic can be 
accomplished. 

3? For the embodiment of the invention using triangulation, it is desirable for the laser diodes, 

scanning laser diode or other light source to be displaced as far as reasonably possible from the camera in 
order to permit the maximum accuracy for the triangulation calculations. In an automobile, as much as six 
inches exists from one side of the exterior rear view mirror to the other side. This is marginal. For large 
trucks, the vertical distance separating the top and bottom of the rear housing can be as much as 24 inches. 
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In both cases, the laser diode would be placed at one extreme and the camera at the other extreme of the 
mirror housing. An alternate approach is to place the camera on the mirror housing but to place the light 
source on the vehicle side. Alternately, both the camera and the light source can be placed at appropriate 
positions on the side of the vehicle. The key is that the direction of the light source should cross field of 

5 view of the camera at at least a 10 degree angle. 

Since the dots or a line created by the light source will be in the infrared spectrum and the 
majority of the light coming from objects in the blind spot will be in the optical spectrum, the possibility 
exists to separate them through the use of an infrared filter which will allow more accurately the 
determination of the location of the reflection from the laser diode onto the optical array. Such filters can 

0 be done either mathematically or through the imposition of a physical filter. However this approach 
requires a mechanical mechanism to move the filter in and out of the camera field of view. Alternately, to 
eliminate the need to move the filter, a pin diode or equivalent dedicated receiver can be used to receive 
the reflected infrared light. 

The blind spot problem for trucks is particularly difficult. Trucks experience the same type of 

5 blind spot as do automobiles where the blind spot extends the length of the vehicle. However, the truck 
driver is also unable to see objects that are in another blind spot extending from forward of the front of the 
vehicle back typically 25 feet. This blind spot has been discussed in greater detail in U.S. Pat. No. 
5,463,384 and International Publication No. WO 90713 103. Trucks also have blind spots behind the trailer 
which are problematic during backup maneuvers. The invention disclosed herein is applicable to all three 

0 blind spot situations for trucks, automobiles or other vehicles. 

It is noteworthy that some trucks have the capability of automatically rotating the side rear view 
mirrors based on the relative angle between the cab and the trailer. Such mirror systems are designed so 
that they maintain their orientation relative to the trailer rather than the cab. The blind spot monitoring 
system of this invention can make appropriate use of this technology to monitor the space along side of the 

5 trailer rather then cab- 
Buses also have a blind spot directly in front vehicle and children have been run over by school 
buses when the driver was not aware that a child was crossing in front of the bus after embarking from the 
bus. The system of this invention is also applicable for monitoring this blind spot and warning a bus driver 
that a child is in tins blind spot. 

0 Naturally, the images obtained from various locations outside of the vehicle can alternately be 

achieved by cameras or by fiber-optic systems. This invention is not limited to the physical placement of 
cameras at particular locations when a fire optic transmission system could be used as well. 

The principles of this invention can also be used for other purposes such as intelligent cruise 
control, speed over ground sensors, parking aids, height sensors for active suspensions., anticipatory crash 

5 sensors and obstacle detection systems. 

Although a trained neural network is contemplated for the preferred embodiments of this 
invention, adaptive neural networks are also applicable especially where the host vehicle may be towing 
different loads that could confuse a static trained system. In this case, part of the system can be made 
adaptive to adjust to the particular load being pulled by the vehicle. 
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OBJECTS AND SUMMARY OF THE INVENTION 

This invention is primarily a system to identify, locate and monitor objects outside of a motor 
vehicle, such as an automobile or truck, by illuminating the objects outside of the vehicle with 
5 electromagnetic radiation, and preferably infrared radiation, or using radiation naturally emanating from 
the object, or reflected from the environment and using one or more lenses to focus images of the contents 
onto one or more arrays of charge coupled devices (CCD's) or CMOS arrays. Outputs from the CCD or 
CMOS arrays, are analyzed by appropriate computational means employing trained pattern recognition 
technologies, to classify, identify and/or locate the external objects. In general, the information obtained 
10 by the identification and monitoring system may be used to affect the operation of at least one other system 
in the vehicle. 

In some embodiments of the invention, several CCD or CMOS arrays are placed in such a manner 
that the position and the motion of an object toward the vehicle can be monitored as a transverse motion 
across the field of the array. In this manner, the need to measure the distance from the array to the object is 

15 obviated. In other embodiments, a source of infrared light is a pulse modulated laser which permits an 
accurate measurement of the distance to the point of reflection through the measurement of the time-of- 
flight of the radiation pulse which may use the technique of range gating. In still other embodiments, a 
scanning array of infrared LEDs are used to illuminate spots on the object in such a manner that the 
location of the reflection of the spot in the field of view provides information as to the location of the 

20 object relative to the vehicle through triangulation. In still other embodiments, a scanning laser diode or 
equivalent is modulated and the distance determined by phase measurement with or without range gating. 
The light can also be modulated with more than one frequency to extend the range without a loss in range 
accuracy at the expense of slightly more complicated electronics and/or software. 

In some applications, a trained pattern recognition system, such as a neural network or neural- 

25 fuzzy system, is used to identify the object in the blind spot of the vehicle. In some of these cases, the 
pattern recognition system determines which of a library of images most closely matches the object in the 
blind spot and thereby the location of the object can be accurately estimated from the matched images and 
the relative size of the captured image thus removing the requirement for special lighting or other distance 
measuring systems. 

30 Principle objects and advantages of the optical sensing system in accordance with the invention 

are: 

1. To recognize the presence of an object in the blind spot on either side, in the front, or the 

rear of a vehicle and to use this information to affect the operation of another vehicle system such as the 
steering wheel, brake system, collision warning system, among others. 
35 2. To recognize the presence of a particular object in a blind spot of a motor vehicle and 

then to determine its position and/or velocity and to use this information to affect the operation of another 
vehicle system. 

3. To determine the position, velocity or size of an object in a blind spot in a motor vehicle 

and to utilize this information to control the velocity or direction of motion of the vehicle.. 
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4. To provide an anticipatory sensor that permits accurate identification of the about-to- 
impact object in the presence of snow and/or fog whereby the sensor is located within the vehicle. 

5. To provide a smart headlight dimmer system which senses the headlights from an 
oncoming vehicle or the tail lights of a vehicle in front of the subject vehicle and identifies these lights 

5 differentiating them from reflections from signs or the road surface and then sends a signal to dim the 
headlights and optionally to notify the exterior monitoring system of the existence and location of these 
other vehicles. 

6. To provide a blind spot detector that detects and categorizes an object in the driver's 
blind spot or other location in the vicinity of the vehicle, and warns the driver in the event the driver begins 

10 to change lanes, for example, or continuously informs the driver of the state of occupancy of the blind spot. 

7. To provide a vehicle exterior monitoring system which has high resolution permitting the 
location of all objects proximate to the vehicle to be accurately determined. 

8. To provide an object vehicle exterior monitoring system which reduces the glare from 
sunlight and headlights by imposing a filter between the eyes of an occupant and the light source and then 

1 5 sends a signal to dim the headlights and optionally to notify the exterior monitoring system of the existence 
and location of these other vehicles. 

9. To provide a camera system for exterior monitoring, which can adjust on a pixel by pixel 
basis for the intensity of the received light. 

10. To use modulated illumination to determine the distance to objects in the blind spot 
20 either at particular representative points ; on a line or on a pixel by pixel basis. 

11. To provide a high dynamic range camera system for exterior monitoring, which permits 
image acquisition during a wide range of ambient lighting conditions permitting its use at night and during 
strong daylight sunlight conditions. 

12. To provide an active pixel camera system for exterior monitoring, which permits image 
25 acquisition during a wide range of ambient lighting conditions permitting its use at night and during strong 

daylight sunlight conditions and also permits the addition of other influencing electronic or optic element 
within the CMOS structure. 

13. To provide a method of obtaining distance measurements to elements of objects exterior 
to the vehicle by using either time-of-flight or modulated light and phase comparison information. 

30 14. To provide a method of obtaining three-dimensional information of objects exterior to 

the vehicle by using either time-of-flight or modulated light and phase comparison information. 

15. To provide a device for obtaining threedimensional information of an object exterior to 
the vehicle by using an electronic shutter or light valve to control the passage of light reflected from the 
object as a function of the phase of such reflected light. 

35 These and other objects and advantages will become apparent from the following description of 

the preferred embodiments of the vehicle exterior identification and monitoring system of this invention. 

In order to achieve one or more of the foregoing objects, an arrangement for obtaining 
information about objects in an environment around a vehicle comprises light emitting means arranged on 
the vehicle for emitting infrared light into the environment around the vehicle, receiver means arranged on 
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the vehicle for receiving infrared hght from the environment around the vehicle and measurement means 
coupled to the light emitting means and the receiver means for measuring time between emission of the 
infrared Hght by the light emitting means and reception of the infrared light by the receiver means. The 
measured time correlates to distance between the vehicle and an object from which the infrared light is 
5 reflected. The light emitting means may comprise an array of laser diodes., a pulsed laser or a continuous 
laser beam directing infrared light in a line and means for controlling the laser beam to change a direction 
of the infrared light such that infrared light traverses a volume of space alongside the vehicle. In the latter 
case, the receiver means could comprise a single pixel receptor. Otherwise, the receiver means may 
comprise a CCD array, a CMOS array, an HDRC camera, a dynamic pixel camera and an active pixel 
10 camera. 

A processor or control circuitry is usually coupled to the receiver means, by the use of wires or 
even wirelessly, for providing an identification of the object from which light is reflected. The processor 
preferably utilizes pattern recognition techniques such as a neural network or even possibly a modular 
neural network to determine the distance between the vehicle and the object, the position of the object 

15 and/or the identify the object from which hght is reflected. The processor can be designed to create a 
three-dimensional map of a portion of the environment surrounding the vehicle based on the received 
optical waves or energy, and then extract features from the three-dimensional map. In the latter case, a 
display is provided in the passenger compartment visible to a driver of the vehicle for displaying features 
or representations derived from features extracted from the three-dimensional map. 

20 As t0 the position of the hght receiving components and associated receivers, they may be 

collocated or spaced apart from one another. When used for blind spot detection, they should be 
positioned around the vehicle to encompass the blind spots of the driver. 

A system for controlling a vehicular system based on the presence of an object in an environment 
around a vehicle comprises any of the foregoing constructions of the arrangement for obtaining 

25 information about an object in the environment surrounding the vehicle. A vehicular system is then 
adapted to be controlled or adjusted upon the determination of the presence of an object in the environment 
around the vehicle. To this end, a processor is coupled to the arrangement and the vehicular system for 
obtaining the information about the object based at least on the infrared light received by the receiver 
means and controlling the vehicular system based on the obtained information. The vehicular system may 

30 be a display visible to a driver of the vehicle for displaying features or representations derived from 
features extracted from a three-dimensional map generated by the processor from the optical waves or 
energy received by the receiver means. The vehicular system could also be a steering wheel having an 
adjustable turning resistance, which is adjustable with a view toward avoiding accidents, and an audio 
alarm and a visual warning viewable by a driver of the vehicle. 

35 A kQ y to one embodiment of this invention is the combination of an optical system with 

superimposed infrared patterns. In other words, the basic system is a passive optical system. Another key 
is the use of a high dynamic range camera that can be used to get the image. A third key is to use 
modulated light or triangulation to determine the distance to an object in the blind spot. 

Another key to this invention is to interpret and identify the image rather than just offering it to 
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the driver. 

Still another key to this invention is to provide methods of obtaining three dimensional 
information about objects in the vicinity of the vehicle by using modulated illumination and phase 
information from the reflected light. 



BRIEF DESCRIPTION OF THE DRAWINGS 

The following drawings are illustrative of embodiments of the invention and are not meant to 
limit the scope of the invention as encompassed by the claims. 

FIG. 1 is a perspective view of an automobile showing a preferred mounting location for the 
10 optical blind spot detection system in accordance with the invention. 

FIG. 2 is a perspective view of the vehicle of FIG. 1 shown operating on a highway. 

FIG. 3 A is a detailed view of an automobile mirror assembly showing the location of a LED array 
or a scanning laser and the CMOS camera. 

FIG. 3B is a detailed view of a truck mirror assembly showing the location of a LED array or a 
15 scanning laser and the CMOS camera. 

FIG. 4 is an overhead view of an alternate blind spot monitoring system where the light source 
and camera are not collocated. 

FIG. 5 is a view similar to FIG. 2 however showing the pattern of laser diode illumination 
projected from a vehicle mirror preferred installation. 
20 FIG. 6 A is a top view of a large truck vehicle showing the coverage of a side blind spot area. 

FIG. 6B is a side view of the large truck vehicle of FIG. 6A showing the coverage of a side blind 
spot area. 

FIG. 7A is a top view illustrating the coverage of the forward right hand side of a truck vehicle 
blind spot. 

25 FIG. 7B is a side view illustrating the coverage of the forward right hand side of a truck vehicle 

blind spot. 

FIG. 8 A is a side view illustrating the application to a bus for monitoring the space in front of the 

bus. 

FIG. 8B is a front view illustrating the application to a bus for monitoring the space in front of the 

30 bus. 

FIG. 9 is a top view of a system applied to monitor the rear of the truck trailer to protect for 
backup accidents. 

FIG. 10 is a block diagram illustrating the blind spot detector, steering control, display and 
warning system. 

35 FIG. 11 shows an icon display for the instrument panel and an alternate heads up display 

indicating the position of the host vehicle and the positions of surrounding potentially threatening vehicles 
as seen from above. 

FIG. 12 is an illustration similar to FIG. 11 showing the projection of the images onto a heads-up 

display. 
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FIG. 13 A illustrates a view of the image as seen by a side rear view camera of FIG. 1. 
FIG. 13B illustrates a view of the image of FIG. 13 A after a stage of image processing. 
FIG 13C illustrates a view of the image as FIG. 13B after the vehicle has been abstracted. 
FIG. 14 illustrates a lane change problem in congested traffic. 

FIG 15 is a perspective view of a vehicle about to impact the side of another vehicle showing the 
location of the various parts of the anticipatory sensor system of this invention. 
FIG. 16 is a flowchart for the lane change problem. 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 

Referring now to the drawings wherein the same reference numerals refer to like elements, a 
perspective semi-transparent view of an automobile is shown generally as 100 in FIG 1. A driver 101 of 
the automobile sits on a seat 102. Five transmitter and/or receiver assemblies 110, 111, 112, 113 and 114 
are positioned at various places with views of the environment surrounding the vehicle. Processor means 
such as control circuitry 120 is connected to the transmitter/receiver assemblies 110-114 by appropriate 
15 wires, not shown, or wirelessly and controls the transmission of waves or energy from the transmitter 
portion of the assemblies and captures the return signals received by the receiver portion of the assemblies. 
Control circuitry 120 usually contains one or more analog to digital converters (ADCs) or frame grabbers, 
a microprocessor containing sufficient memory and appropriate software including pattern recognition 
algorithms, and other appropriate drivers, signal conditioners, signal generators, etc. Usually, in any given 
20 implementation, only two to four of the transmitter/receiver assemblies would be used. These optical 
transmitter/receiver assemblies, also referred to herein as transducer assemblies, are comprised of an 
optical transmitter or light emitting component, which may be an infrared LED, a laser with a diverging 
lens, a floodlight or a scanning laser assembly, any of which can be modulated, and a receiver such as a 
CCD or CMOS array or pin diode or equivalent. Preferably, a transmitter/receiver assembly comprises an 
25 active pixel CMOS array or an HDRC array as discussed below. The transducer assemblies map the 
location of the objects and features thereof, in a two and/or three-dimensional image as will also be 
described in more detail below. 

The foregoing examples of possible wave/energy/light emitting components and 
light/wave/energy receiver components are not intended to limit the invention and it should be understood 

30 by those skilled in the art that other transmitter and receiver components and combinations can be used in 
accordance with the invention without deviating from the scope and spirit thereof. 

In a preferred embodiment four transducer assemblies axe positioned around the exterior of the 
vehicle in the spaces to be monitored, each comprising one or more LEDs or scanning laser diodes and a 
CMOS array with a light valve and an appropriate lens. Although illustrated together, the illuminating 

35 source will frequently not be co-located with the receiving array particularly when triangulation distance 
measurement is used, as described in more detail below. The LED or laser emits a controlled angle 
diverging beam of infrared radiation that illuminates a particular space and illuminates an object at a 
particular point that depends on die location of the object relative to the vehicle and the direction of the 
LED or laser beam. 
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The image from each array is used to capture two or three dimensions of object position 
information, thus, the array of assembly 110, which is located approximately behind the driver's door on 
the B-pillar provides both vertical and transverse information on the location of an object in the vicinity of 
the vehicle. A similar view from a location on the passenger side is obtained from the array of assembly 
5 ill. The mounting locations of the assemblies 110,111 shown in FIG. 1 are exemplary and are not 
intended to limit the possible positions for placement of the assemblies. Other positions for installation of 
the assemblies on the sides of the vehicle is contemplated by the inventors. For example, the assemblies 
could be placed on the side of the vehicle alongside the passenger compartment, engine compartment or 
trunk compartment. 

10 If the receiving array of assembly 110 contains a matrix of 100 by 100 pixels, then 10,000 pixels 

or data elements of information will be created each time the system interrogates the space on the driver 
side of the vehicle, for example. Interrogation of the space on the driver side of the vehicle would entail 
commanding the assembly to transmit optical waves or energy into the environment surrounding the 
vehicle by means of the transmitter component of the assembly 110 and receiving any reflected optical 

1 5 waves or energy by the receiver component of the assembly 1 10. There are many pixels of each image that 
can be eliminated because they do not contain any useful information. This typically includes the corner 
pixels and other areas where an object cannot reside. This pixel pruning can typically reduce the number 
of pixels by up to 20 percent resulting in approximately 8,000 remaining pixels. The output from each 
array is then preprocessed to extract the salient features and fed to an artificial neural network, or other 

20 pattern recognition system, to identify of the object. This preprocessing frequently makes use of distance 
information to separate one object from another and from other parts of the scene. Once this operation is 
completed for all of the object images, the identification of the objects in the space proximate to the driver 
side of the vehicle has been determined. 

A pattern recognition technique such as a trained neural network can be used to determine which 

25 of the trained occupancies most closely corresponds to the measured data. The output of the neural 
network can be an index of the setup that was used during training that most closely matches the current 
measured state. This index can be used to locate stored information from the matched trained occupancy. 
Information that has been stored for the trained occupancy typically includes the locus of the centers of 
different objects and an appropriate icon. For the case of FIG. 1, it is also known from one of the 

30 techniques to be described below where the object is located relative the vehicle. 

There are many mathematical techniques that can be applied to simplify the above process. One 
technique used in military pattern recognition, for example, uses the Fourier transform of particular areas 
in an image to match by correlation with known Fourier transforms of known images. In this manner, the 
identification and location can be determined simultaneously. There is even a technique used for target 

35 identification whereby the Fourier transforms are compared optically. Other techniques utilize 
thresholding to limit the pixels that will be analyzed by any of these processes. Still other techniques 
search for particular features and extract those features and concentrate merely on the location of certain of 
these features. A particularly useful technique calculates the location of the edges of the object in the blind 
spot and uses these extracted edges as the features that are fed to the neural network. (See, for example.. 
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the Kage et al artificial retina paper referenced below.) 

The principle used in this preferred embodiment of the invention is to use images of different 
views of an object in the .blind spot to correlate with known images that were used to train a neural 
network for blind spot occupancy. Then, carefully measured positions of the known images are used to 
5 locate particular parts of the object such as the windshield, tires, radiator grill, headlights, etc. An alternate 
approach is to make a three-dimensional map of the object in the blind spot based on the optical energy or 
waves received by the receiver components of the assemblies 110-114 and to precisely locate these 
features using neural networks, fuzzy logic or other rules. One method of obtaining a three-dimensional 
map is to utilize a scanning laser radar system where the laser is operated in a pulse mode and the distance 

10 from the object being illuminated is determined using range-gating in a manner similar to that described in 
various patents on micropower impulse radar to McEwan. (See, for example, U.S. Pat. Nos. 5,457,394 and 
5,521,600). Alternately, the laser can be modulated and the phase of the reflected and the transmitted light 
can be compared to determine the distance to the object. 

The scanning portion of the laser radar device can be accomplished using rotating mirrors, 

15 mechanical motors, or preferably, a solid state system, for example one utilizing Te0 2 as an optical 
diffraction crystal with lithium niobate crystals driven by ultrasound (although other solid state systems not 
necessarily using Te0 2 and lithium niobate crystals could also be used). An alternate method is to use a 
micromachined mirror, which is supported at its center and caused to deflect by miniature coils or 
electrostatically. Such a device has been used to provide two-dimensional scanning to a laser. This has 

20 the advantage over the Te0 2 - lithium niobate technology in that it is inherently smaller and lower cost and 
provides two-dimensional scanning capability in one small device. The maximum angular deflection that 
can be achieved with this process is in the order of about 10 degrees. A diverging lens or mirror can be 
used to achieve a greater angular scan if necessary. A preferred approach is to use passive optical images 
with superimposed infrared dots created by an array of infrared laser diodes in a manner similar to that 

25 described in U.S. Pat. No. 6,038,496 discussed above. 

An alternate method of obtaining three-dimensional information from a scanning laser system is 
to use multiple arrays to replace the single arrays used in FIG. 1 . In the case, the arrays are displaced from 
each other and, through triangulation, the location of the reflection from the illumination by a laser beam 
of a point on the object can be determined by triangulation and/or correlation in a manner that is 

30 understood by those skilled in the art. 

One important point concerns the location and number of optical assemblies. For an automobile 
one assembly is generally placed on each side of the vehicle 110, 111. In some embodiments, a third 
assembly 1 12 can be placed to view the blind spot behind the vehicle and a fourth 113 can be placed to 
view in front of the vehicle for cruise control. 

35 An alternate configuration is shown at 1 14 which is a lens arrangement which provides a view of 

360 degrees by approximately 20 degrees. Although this camera does not provide as complete a view of 
objects in the various blind spots, it is possible using a single device to observe areas on both sides as well 
as the front and back of the vehicle. The same lens is used for receiving the images and for protecting a 
rotating scanning laser beam which approximately bisects the 20 degree angle. This rotating laser beam is 
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modulated thereby permitting the distance to the reflected laser light to be determined. The returned laser 
light is captured by a rotating mirror which also serves to deflect the laser beam. This mirror is positioned 
so that it is above the portion of the lens used for receiving the images such that laser system does not 
interfere with the imaging system. Special lenses are used to collect the light from the spherical segmented 
5 lens and project the combined image onto the CMOS imager. In some cases, software is provided to 
remove known distortions for image analysis or, in other cases, this is not necessary as the pattern 
recognition system has been trained on the combined received image, or a segmented version thereof 
which divides the image into, for example, four segments representing front, right, rear, and left quadrants. 
The particular locations of the optical assemblies are selected to provide accurate information as 

10 to the locations of objects in the blind spots. This is based on an understanding of what information can be 
best obtained from a visual image. There is a natural tendency on the part of humans to try to gauge 
distance from the optical sensors directly. This typically involves focusing systems, stereographic 
systems, multiple arrays and triangulation, time-of-flight measurement, phase comparison, etc. What is 
not intuitive to humans is to not try to obtain this distance directly from apparatus or techniques associated 

15 with the mounting location. Whereas ultrasound is quite good for measuring distances from the transducer 
(the z-axis), optical systems are better at measuring distances in the vertical and lateral directions (the x 
and y-axes). For monitoring the interior of the vehicle, such as described in U.S. patent application Ser. 
No. 09/476,255 filed December 30 1999, this can more easily done indirectly by another transducer. That 
is, the z-axis to one transducer is the x-axis to another. For external monitoring, the preferred approach, as 

20 described below, is to use an array of LEDs or a scanning laser and locate the position of the object in 
blind spot by triangulation, time-of-flight or phase measurement although sometimes appropriately located 
cameras in concert can provide three-dimensional information directly. 

Systems based on ultrasonics and neural networks, and optics and optical correlation have been 
very successful in analyzing the seated state of both the passenger and driver seats in the interior of 

25 automobiles. Such systems are now going into production for preventing airbag deployment when a rear 
facing child seat or an out-of-position occupant is present. The ultrasonic systems, however, suffer from 
certain natural limitations that prevent the system accuracy from getting better than about 99 percent. 
These limitations relate to the fact that the wavelength of ultrasound is typically between 3 and 8 mm. As 
a result unexpected results occur which are due partially to the interference of reflections from different 

30 surfaces. Additionally, commercially available ultrasonic transducers are tuned devices that require 
several cycles before they transmit^ significant energy and similarly require several cycles before they 
effectively receive the reflected signals. This requirement has the effect of smearing the resolution of the 
ultrasound to the point that for example, using a conventional 40 kHz transducer, the resolution of the 
system is approximately three inches, although this has been recently improved. These limitations are also 

35 present in the use of ultrasound for exterior vehicle monitoring. 

In contrast, the wavelength of the portion of the infrared spectrum that is contemplated for 
preferred use in the invention is less than five microns and no significant interferences occur. As a result, 
resolution of the optical system is determined by the pixel spacing in the CCD or CMOS arrays or the 
speed of the pin diode and scanner when used. For this application, typical arrays have been selected to be 
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100 pixels by 100 pixels and therefore, the space being imaged can be broken up into pieces that are 
significantly less than a few inches in size. Naturally, if greater resolution is required arrays having larger 
numbers of pixels are readily available. Another advantage of optical systems is that special lenses can be 
used to magnify those areas where the information is most critical and operate at reduced resolution where 
5 this is not the case. For example, the area closest to the center of the blind spot can be magnified and those 
areas that fall out of the blind spot but are still being monitored can be reduced- This is not possible with 
ultrasonic or radar systems where it is even very difficult to get an image of sufficient resolution to permit 
an identification of the object to be accomplished. 

Additional problems of ultrasonic systems arise from the slow speed of sound and diffraction 
10 caused by variations in air density. The slow sound speed limits the rate at which data can be collected and 
thus eliminates the possibility of tracking the motion of an object moving at high speed relative to the 
vehicle. 

In the case of FIG. 1, transmitter/receiver assemblies 1 10-1 14 emit infrared waves that reflect off 
of objects in the blind spot and return thereto. Periodically, the device, as commanded by control circuit 

15 120, transmits a pulse of infrared waves and the reflected signal is detected by a different device. 
Alternately a continuous scanning arrangement can be used. The transmitters can either transmit 
simultaneously or sequentially. An associated electronic circuit and algorithm in control circuit 120 
processes the returned signals as discussed above and determines the identity and location of the object in 
the blind spot. This information is then sent to a warning system that alerts the driver to the presence of the 

20 object as described in more detail below. Although a driver side system has been illustrated, a similar 
system is also present on the passenger side and can be applied to the front and rear of the vehicle. 

The accuracy of the optical sensor is dependent upon the accuracy of the camera. The dynamic 
range of light external to a vehicle exceeds 120 decibels. When a car is driving at night, for example, very 
little light may be available whereas when driving in a bright sunlight, the light intensity can overwhelm 

25 most cameras. Additionally, the camera must be able to adjust rapidly to changes and light caused by, for 
example, the emergence of the vehicle from a tunnel, or passing by other obstructions such as trees, 
buildings, other vehicles, etc. which temporarily block the sun and cause a strobing effect at frequencies 
approaching 1 kHz. 

Recently, improvements have been made to CMOS cameras that have significantly increased their 
30 dynamic range. New logarithmic high dynamic range technology such as developed by IMS Chips of 
Stuttgart, Germany, is now available in HDRC (High Dynamic Range CMOS) cameras. This technology 
provides a 120 dB dynamic intensity response at each pixel in a monochromatic mode. The technology 
thus has a 1 million to one dynamic range at each pixel. This prevents blooming, saturation and flaring 
normally associated with CMOS and CCD camera technology. This solves a problem that will be 
35 encountered in an automobile when going from a dark tunnel into bright sunlight. 

There is also significant infrared radiation from bright sunlight and from incandescent lights. 
Such situations may even exceed the dynamic range of the HDRC camera and additional filtering may be 
required. Changing the bias on the receiver array, the use of a mechanical iris, light valve, or of 
electrochromic glass or liquid crystal or similar filter can provide this filtering on a global basis but not at a 
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also significantly reduce the amount of training effort and associated data collection that must be 
undertaken during the development of the neural network algorithm or other computational intelligence 

system. Note that in some applications, it is possible to use other more deterministic image processing or 

-«* * 

pattern recognition systems than neural networks such as optical correlation techniques. 
5 Another important feature of the HDRC camera from IMS Chips is that the shutter time is 

constant at less than 100ns irrespective of brightness of the scene. The pixel data arrives at constant rate 
synchronous with the internal imager clock. Random access to each pixel facilitates high-speed intelligent 
access to any sub-frame (block) size or sub-sampling ratio and a trade-off of frame speed and frame size 
therefore results. For example, a scene with 128 K pixels per frame can be taken at 120 frames per second, 

1 0 or about 8 milliseconds per frame, whereas a sub-frame can be taken at as high as 4000 frames per second 
with 4 K pixels per frame. This combination allows the maximum resolution for the identification and 
classification part of the object sensing problem while permitting a concentration on those particular pixels 
which track the leading edge of the object for dynamic position tracking. In fact, the random access 
features of these cameras can be used to track multiple parts of the image and thus in some cases multiple 

15 objects simultaneously while ignoring the majority of the image, and do so at very high speed. For 
example, several motorcycles or pedestrians in the blind spot can be tracked simultaneously by defining 
separate sub-frames for each object that need not be connected. This random access pixel capability, 
therefore, is optimally suited for recognizing and tracking multiple objects in the blind spot. It is also 
suited for monitoring the environment outside of the vehicle other than for the purpose of blind spot 

20 detection such as collision avoidance and anticipatory sensing. Photobit Corporation of 135 North Los 
Robles Ave., Suite 700, Pasadena, California 91101 manufactures another camera with some 
characteristics similar to the IMS Chips camera. Other competitive cameras can be expected to appear on 
the market. 

Photobit refers to their Active Pixel Technology as APS. According to Photobit, in the APS, both 
25 the photodetector and readout amplifier are part of each pixel. This allows the integrated charge to be 
converted into a voltage in the pixel that can then be read out over X-.Y wares instead of using a charge 
domain shift register as in CCDs. This column and row addressability (similar to common DRAM) allows 
for window of interest readout (windowing) which can be utilized for on chip electronic pan/tilt and zoom. 
Windowing provides added flexibility in applications, such as disclosed herein, needing image 
30 compression, motion detection or target tracking. The APS utilizes intra-pixel amplification in 
conjunction with both temporal and fixed pattern noise suppression circuitry (i.e., correlated double 
sampling), which produces exceptional imagery in terms of wide dynamic range (-75 dB) and low noise 
(-15 e- rms noise floor) with low fixed pattern noise (<0.15% sat). Unlike CCDs, the APS is not prone to 
column streaking due to blooming pixels. This is because CCDs rely on charge domain shift registers that 
35 can leak charge to adjacent pixels when the CCD register overflows. Thus, bright lights "bloom 11 and cause 
unwanted streaks in the image. The active pixel can drive column busses at much greater rates than 
passive pixel sensors and CCDs. On-chip analog-to-digital conversion (ADC) facilitates driving high speed 
signals off chip. In addition, digital output is less sensitive to pickup and crosstalk, facilitating computer 
and digital controller interfacing while increasing system robustness. A high speed APS recently 
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developed for a custom binary output application produced over 8,000 frames per second, at a resolution of 
128 x 128 pixels. It is possible to extend this design to a 1024 x 1024 array size and achieve greater than 
1000 frames per second for machine vision. All of these features are important to many applications of 
this invention. U.S. Pat. No. 5,471,5 15 provides additional information on the APS camera from Photobit. 
5 To put this into perspective, a vehicle passing another vehicle at a relative velocity of 60 mph moves 
approximately 1 inch per millisecond relative to the slower vehicle. This renders the frame rate and 
computational times critically important and within the capabilities of the HDRC and APS technologies. 

These advanced cameras, as represented by the HDRC and the APS cameras, now make it 
possible to more accurately monitor the environment in the vicinity of the vehicle. Heretofore, the large 

10 dynamic range of environmental light has either blinded the cameras when exposed to bright light or else 
made them unable to record images when the light level was low. Even the HDRC camera with its 120 dB 
dynamic range may be marginally sufficient to handle the fluctuations in environmental light that occur. 
Thus, the addition of an electrochromic, liquid crystal, light valve or other similar filter may be necessary. 
This is particularly true for cameras such as the Photobit APS camera with its 75 dB dynamic range. 

15 At about 120 frames per second, these cameras are adequate for cases where the relative velocity 

between vehicles is low. There are many cases, however, where tins is not the case and a higher 
monitoring rate is required. This occurs for example, in collision avoidance and anticipatory sensor 
applications as well as in blind spot applications where one vehicle is overtaking another at high speed. 
The HDRC camera is optimally suited for handling these cases since the number of pixels that are being 

20 monitored can be controlled resulting in a frame rate as high as about 4000 frames per second with a 
smaller number of pixels. 

Another key advantage of the HDRC camera is that it is quite sensitive to infrared radiation in the 
0.8 to 1 micrometer wavelength range. This range is generally beyond visual range for humans permitting 
this camera to be used with illumination sources that are not visible to the human eye. Naturally, a notch 

25 frequency filter is frequently used with the camera to eliminate unwanted wavelengths. These cameras are 
available from the Institute for Microelectronics (IMS Chips), Allamndring 30a, D-70569 Stuttgart, 
Germany with a variety of resolutions ranging from 512 by 256 to 720 by 576 pixels and can be custom 
fabricated for the resolution and response time required. 

FIG. 2 illustrates the arrangement of FIG. 1 in a traffic situation. Optical assembly 111 on the 

30 "host" vehicle contains the illuminating light source and the CMOS array. This illuminating light source, 
either through an array of scanning LEDs or a scanning laser radar device, distributes infrared radiation or 
energy in the form of distinct narrow angle beams or a line that covers or fills in the blind spot between 
bounding lines 117 and 118. Any object such as vehicle 200 which is within tins blind spot will be 
illuminated by infrared and the image of object will be captured by the CMOS array. 

35 An optical infrared transmitter and receiver assembly is shown generally at 1 12 in FIG. 3 A and is 

mounted onto the side rear view mirror 135. Assembly 112, shown enlarged, comprises a source of 
infrared radiation including an array of 20 infrared LEDs, shown generally at 131, and a CCD or CMOS 
array 132 of typically 160 pixels by 160 pixels. In this embodiment, a t: heads-up" display can be used to 
show an artificial image to the driver showing the host vehicle and objects that are in the blind spot as 
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described below. 

If two spaced apart CCD arrays are used, then the distance to the various objects within the blind 
spot can be found by using a triangulation algorithm which locates similar features on both images and 
deterrnines their relative location on the images. This is frequently referred to as a stereoscopic system 
5 such as described in European Patent Application No. EP 0885782 Al. An alternate method is to use a lens 
with a short focal length. In this case, the lens is mechanically focused to determine the clearest image and 
thereby obtain the distance to the object This is similar to certain camera auto-focusing systems such as 
one manufactured by Fuji of Japan. Naturally, other methods can be used as described in the patents and 
patent applications referenced above. 

10 FIG. 3B shows a similar arrangement for mounting on a truck mirror. In this case, since the 

geometry of the mirror provides greater separation vertically than horizontally, the Ulumination source 13 1 
is placed on the top of the mirror housing and the imager 132 at the bottom. The "imager" may comprise a 
CCD array or CMOS array. 

Once a vehicle exterior monitoring system employing a sophisticated pattern recognition system, 

15 such as a neural network or optical correlation system, is in place, it is possible to monitor the motions of 
the object over time, and thereby determine if the object is acting in a predictable manner. If not, the 
driver of the host vehicle can be warned so that he/she can take evasive action. For example, a vehicle 
may be in the blind spot and the driver may be losing control of the vehicle as may happen in a passing 
situation when the passing vehicle has hit a patch of ice. This warning may be sufficient to allow the 

20 driver of the host vehicle to slow down and thereby avoid an accident with the out-of-control vehicle. The 
system can also be used to turn on the vehicle hazard lights, sound the horn or take other appropriate action 
in case the driver of the threatening vehicle has fallen asleep and to warn other adjacent vehicles of a 
potentially dangerous situation. Thus, in general, another vehicular system can be controlled based on the 
determination of the motion of the object detected in the blind spot. The use of a heads-up display is 

25 particularly useful for such a warning system since the driver is presumably looking through the 
windshield. Out-of-control monitoring can also apply to the host vehicle if its trajectory is unexpected 
relative to objects along the roadside or other proximate vehicles. 

Infrared waves are shown coming from the side transducer assemblies 140 and 143 in FIG. 4. In 
this case, the CMOS imagers 145 and 146 are mounted on the side rear view mirrors providing ample 

30 displacement for triangulation calculations. Thus, FIG. 4 shows one arrangement of non-collocated 
transmitters and receivers, it being understood that other arrangements in which the transmitters are not 
collocated with the receivers are also within the scope and spirit of the invention. 

FIG. 5 illustrates two optical systems each having a source of infrared radiation and a CCD or 
CMOS array receiver. In this embodiment, transducers 1 10 and 11 1 are CMOS arrays having 160 by 160 

35 pixels covered by a lens. The lens is carefully designed so that it completely covers the blind spot area 
under surveillance. One such sensor placed by the left outside mirror where it can monitor the entire 
vehicle left exterior blind spot with sufficient resolution to determine the occupancy of the blind spot. 
CCD's such as those used herein are available from Marshall Electronics Inc. of Culver City, CA. The 
lens need not be non-distorting. The distortion of a lens can be designed by modifying the shape of the 
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lens to permit particular portions of the exterior passenger compartment to be observed. The particular 
lens design will depend on the location on the vehicle and the purpose of the particular receiver. In this 
example, the light source, which is an array of modulated LCDs is collocated with the CMOS imager. 
Note that although only four beams are illustrated on each side of the vehicle, typically twenty such beams 
5 are used. A modulated scanning laser can alternately be used. 

CCD arrays are in common use in television cameras, for example, to convert an image into an 
electrical signal. For the purposes herein, a CCD will be defined to include all devices, including CMOS 
arrays. APS arrays, TFA arrays, artificial retinas and particularly HDRC and APS arrays, which are 
capable of converting light frequencies, including infrared, visible and ultraviolet, into electrical signals. 

10 The particular CCD array used for many of the applications disclosed herein is implemented on a single 
chip that is less than two centimeters on a side. Data from the CCD array is digitized and sent serially to 
an electronic circuit (at times designated 120 herein) containing a microprocessor for analysis of the 
digitized data. In order to minimize the amount of data that needs to be stored, initial processing of the 
image data can take place as it is being received from the CCD array. In some cases, some image 

1 5 processing can take place on the chip such as described in the Kage et al. artificial retina article referenced 
above. 

One method of determining distance to an object directly without resorting to range finders, 
requiring multiple arrays, is to use a mechanical focusing system. However, the use of such an apparatus 
is cumbersome, expensive, and slow and has questionable reliability. An alternative is to use the focusing 
20 systems described in U.S. Pat. Nos. 5,193,124 and 5,003,166. However, such systems require expensive 
hardware and/or elaborate algorithms and again are slow. 

Another alternative is where an infrared source having a wide transmission angle such that the 
entire contents of the blind spot illuminated, a sort of infrared floodlight. The receiving CCD transducers 
can be spaced apart so that a stereographic analysis can be made by the control circuitry 120. This 
5 circuitry 120 contains a microprocessor with appropriate pattern recognition algorithms along with other 
circuitry as described above. In this case, the desired feature to be located is first selected from one of the 
two returned images from either of the CCD transducers. The software then determines the location of the 
same feature, through correlation analysis or other methods, on the other image and thereby, through 
analysis familiar to those skilled in the art, determines the distance of the feature from the transducers. 
0 Transducers 110 and 111 are illustrated mounted onto the side mirrors of the vehicle, however, 

since these transducers are quite small, typically approximately 2 cm on a side, they could alternately be 
mounted onto the side of the vehicle or many other locations which provides a clear view of the blind spot. 

A new class of laser range finders has particular application here. This product, as manufactured 
by Power Spectra, Inc. of Sunnyvale, California, is a GaAs pulsed laser device which can measure up to 30 
5 meters with an accuracy of < 2 cm and a resolution of < 1 cm. This system can be implemented in 
combination with transducer 1 10 or 1 1 1. Once a particular feature of an object in the blind spot has been 
located, this device can be used in conjunction with an appropriate aiming mechanism to direct the laser 
beam to that particular feature. The distance to diat feature is then known to within 2 cm and with 
calibration even more accurately. 
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In addition to measurements within the blind spot this device has particular applicability in 
anticipatory sensing applications exterior to the vehicle. An alternate technology using range gating or 
phase measurements to measure the time-of-flight of electromagnetic pulses with even better resolution 
can be implemented based on the teaching of the McEwan patents listed above or by modulation of the 
5 laser beam and using phase measurements such as disclosed in U.S. Pat. No. 5,653,462. 

FIG. 6A is an overhead view and FIG. 6B a side view of a truck showing preferred mounting 
locations of optical exterior vehicle monitoring sensors (transmitter/receiver assemblies or transducers) 
160,161. In a typical device, the diameter of the lens is approximately 2 cm and it protrudes from the 
mounting surface by approximately 1 cm. This small size renders these devices almost unnoticeable by 
10 observers exterior to the vehicle. Since these sensors are optical, it is important that the lens surface 
remains relatively clean. Control circuitry 120 contains a self-diagnostic feature where the image returned 
by a transducer is compared with a stored image and the existence of certain key features is verified. If a 
receiver fails this test., a warning is displayed to the driver that indicates that cleaning of the lens surface is 
required. 

15 The truck system shown in FIGS. 6 A and 6B illustrates the use of a single blind spot detection 

system for the entire length of truck. The fundamental issue that determines the size of the blind spot that 
can be monitored with a single system relates to the ability to measure the location of the object. When a 
HDRC camera is used, if an object can seen in the blind spot by the human eye then the camera should 
also be able to obtain a reasonable image. At night, this would require that the object in blind spot have 

20 some form of attached illumination. On a dark cloudy night, the human eye has trouble seeing a car 
parked along the roadway with its lights extinguished. Naturally, the more distant the object, the more 
difficult it is to obtain a recognizable image if illumination is not present. A significant improvement to 
the situation occurs if the blind spot is flooded even with low-level infrared radiation. This argues for an 
infrared floodlight in addition to the distance measuring infrared system. If an infrared floodlight is used 

25 along with multiple cameras displaced from one another, then the location of object in the blind spot can 
be determined by optical correlation between the two images and by triangulation calculations. This may 
be a practical solution for trucks especially those containing multiple trailers. 

The other limiting case is when bright sunlight is present and only a single imager is used for a 
particular blind spot. For this case, a scanning laser infrared beam can still be distinguished as a reflection 

30 off of an object in the blind spot providing a narrow notch filter is used to eliminate all frequencies other 
than particular infrared frequency used. Even in this case, the distance where the reflected infrared beam 
can be ascertained in bright sunlight is limited to perhaps fifteen meters. Therefore this system is marginal 
for long trucks. 

From the above discussion, it would appear that multiple cameras are the only viable solution for 
35 long trucks. A further problem arises in this system design in that if the cameras are located on different 
trailers, or for some other reason can move relative to each other, then the analysis computer must know 
the location and orientation of each of the cameras. There are a variety of ways of accomplishing this 
orientation such as through locating laser beams or monitoring the relative positions of the various 
complements of the truck. In one example, a laser beam is used to illuminate a spot on the road that can be 
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observed from multiple camera locations. Using the position of this reflected dot in the images acquired 
by various cameras, the relative orientation is approximately determined. Naturally, more complicated and 
sophisticated systems are possible. 

The blind spot monitoring systems described above in FIGS. 6A and 6B are mainly applicable 
5 for blind spots occurring during highway travel. For urban travel of a truck where frequent turns are made, 
another blind spot occurs on right hand side of vehicle and extends somewhat forward of the vehicle and 
back somewhat beyond vehicle cab. This area which cannot be seen by the driver can contain pedestrians, 
small vehicles, bicycles, motorcycles, as well as a variety of other objects. Another more local blind spot 
system that covers this area is therefore necessary, as illustrated in FIGS. 7 A and 7B and which is 
10 designated 163. 

The applications described herein have been illustrated mainly using the driver side of the vehicle. 
The same systems of determining the position of an object in the blind spot are also applicable on the 
passenger side. 

A significant number of children are killed every year by being run over by school buses. This 
15 tragic accident occurs when a child leaves the school bus and walks in front of the bus in the driver's blind 
spot. Driver and starts driving the bus and strikes the child. A blind spot monitor of this invention is 
shown mounted on the front of school bus near the top of the engine compartment 180 in FIGS. 8 A and 
8B. This monitoring system alerts the driver of the presence of an object obstructing the path of the school, 
bus. 

20 The use of trainable pattern recognition technologies such as neural networks is an important part 

of this invention, although other non-trained pattern recognition systems such as fuzzy logic, correlation, 
Kalman filters, and sensor fusion can also be used. These technologies are implemented using computer 
programs to analyze the patterns of examples to determine the differences between different categories of 
objects. These computer programs are derived using a set of representative data collected during the 

25 training phase, called the training set. After training, the computer programs output computer algorithms 
containing the rules permitting classification of the objects of interest based on the data obtained after 
installation on the vehicle. These rules, in the form of an algorithm^ are implemented in the system that is 
mounted onto the vehicle. The determination of these rules is important to the pattern recognition 
techniques used in this invention. 

30 Artificial neural networks using back propagation are thus far the most successful of the rule 

determination approaches, however, research is underway to develop systems with many of the advantages 
of back propagation neural networks, such as learning by training, without the disadvantages, such as the 
inability to understand the network and the possibility of not converging to the best solution. In particular, 
back propagation neural networks will frequently give an unreasonable response when presented with data 

35 than is not within the training data, it is well known that neural networks are good at interpolation but 
poor at extrapolation. A combined neural network fuzzy logic system, on the other hand, can substantially 
solve this problem. Additionally, there are many other neural network systems in addition to back 
propagation. In fact, one type of neural network may be optimum for identifying the contents of the blind 
spot and another for determining the location of the object d}Tiamically. 
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The discussion thus far has identified pattern recognition systems and particularly neural network 
pattern recognition systems to be used to identify the contents of the blind spot. One particular neural 
network architecture has been particularly successful in this field. This is known as modular neural 
networks. The concept behind modular neural networks is that when a complicated task is to be 
5 accomplished by a neural network, significant improvements in speed and accuracy can sometimes be 
obtained if the overall problem is divided into a number of smaller problems. A separate neural network is 
then assigned each sub-task. Thus, a network of neural networks is created. 

When a human observes a tree the human mind concentrates on characteristics of that tree and not 
on characteristics of an automobile. Thus the human rnind appears to operate also as a modular neural 
1 0 network. There are many ways of applying this concept to blind spot monitoring. Since both the identity 
and the location of object in the blind spot are to be determined, it is logical to therefore separate the 
problem into a first network that determines the identity of the object and then a variety of additional 
networks that given the identity of the object, deterrnines its location. In addition, a separate neural 
network may be trained to segregate any unknown objects from data that are not understood by the neural 
15 networks because nothing similar was a part of the training database. Additional tasks that can be 
allocated to specific neural networks is to determine environment that the vehicle is operating in. 
Obviously, an automobile in a blind spot looks considerably different at night with its headlights on than in 
bright sunlight. The identification and also the position determining tasks can be more accurate if they are 
segregated by lighting conditions. Similarly, the presence of fog, rain, snow, soiled lenses, and other 
20 factors can have a significant effect on the system accuracy and allocated to separate groups of neural 
networks. 

In some embodiments of this invention, the rules are sufficiently obvious that a trained researcher 
can look at the returned optical signals and devise an algorithm to make the required determinations. In 
others, artificial neural networks are frequently used to determine the rules. One such set of neural 

25 network software for determining the pattern recognition rules, is available from the NeuralWare 
Corporation of Pittsburgh, Peruisylvania and another from International Scientific Research in Rochester 
Hills, Michigan. Numerous books and articles, including more than 500 U.S. patents, describe neural 
networks in great detail and thus the theory and application of this technology is well known and will not 
be repeated here. Neural networks are now beginning to gain more widespread use in the automotive 

30 industry including their use for engine control., occupant spatial sensing for the control of airbags.. side and 
frontal crash sensor algorithms and vehicle diagnostic systems. 

The system generally used in this invention for the determination of the presence of an object in 
the blind spot is the artificial neural network or a neural-fuzzy system. In this case, the network operates 
on the returned signals from the CCD or CMOS array as sensed by transducers such as 1 10, 1 1 1, 1 13 and 

35 114 in FIG. 1, for example. For the case of the left blind spot, through a training session, the system is 
taught to differentiate between many cases including automobiles, pedestrians, bicvcles. trucks, animals 
motorcycles, fences., guard rails., parked vehicles etc. This is done by conducting a large number of 
experiments where data from each of these objects is captured in a variety of positions, velocities and 
vehicle operating conditions (rain., night, bright sunlight, rural roads, interstate highways etc.). As many 
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as 1,000,000 such experiments are run before the neural network is sufficiently trained and validated so 
that it can differentiate among the various cases and output the correct decision with a very high accuracy. 

Once the network is determined, it is possible to examine the result to determine, from the 
algorithm created by the neural network algorithm generating software, the rules that were finally arrived 
5 at by the trial and error training technique. In that case, the rules can then be programmed into a 
microprocessor. Alternately, a neural computer can be used to implement the network directly. In either 
case, the implementation can be carried out by those skilled in the art of pattern recognition using neural 

« 

networks. 

Many systems are now on the market that monitor obstructions in the rear of a vehicle and warn 
10 the driver of the existence of such obstructions when the driver is backing a vehicle. The technologies 
currently used include radar, ultrasound and TV cameras. Neither radar nor ultrasound are capable of 
identifying the object and most such systems cannot locate the object which might allow the driver to 
slightly change his or her direction and avoid a curb or pole, for example. The television camera systems 
typically do not have illumination sources and at best produce a poor television image to the driver that is 
15 difficult to see in sunlight 

The system shown in FIG. 9 illustrates a blind spot monitoring system 190 built according to the 
teachings of this invention. The system utilizes a high dynamic range camera, identification and ranging 
. capability. The view provided to the driver shows the location, size and identity of all objects that are 
within the path of the backing vehicle. The display provides maximum contrast by using icons to represent 
20 the host vehicle and the objects in the blind spot. Naturally although this is shown for a truck, it is equally 
applicable for other vehicles including buses and automobiles. It can also be used in a rear impact 
anticipatory sensor. 

FIG. 10 illustrates a block diagram showing interface between five blind spot monitoring systems 
and control circuitry 120. The control circuitry 120 monitors the output from the five blind spot 

25 monitoring systems and creates icons and places the icons on a display 140,145 which shows the host 
vehicle and all objects in the immediate vicinity of the host vehicle. Software is provided in the 
microprocessor to sound a warning signal 146 under a predetermined set of circumstances such as an 
attempt by die driver to change lanes into a lane occupied by an object in the blind spot. This warning 
signal may also be activated if the driver activates the turn signal. In addition to the audio warning signal, 

30 a visual flashing signal provided on the display and a vibration or pressure or torque or other haptic signal 

* 

applied to the steering wheel to prevent or make it more difficult for driver execute the maneuver. 

FIG. 11 illustrates control module 170 which contains a variety of electronic components 172- 
178. The control module is connected to the blind spot monitors by wires, not shown, or wirelessly and in 
turn it connects to a display on the instrument panel 145 or a heads-up display 140. Based on the 
35 calculations performed in a microprocessor 177, the control module 170 creates die icons on displays 140 
and 145 and additionally initiates audio and haptic warnings as described above. 

FIG. 12 is a further illustration of the heads-up display 140 shown in FIG. 11. The heads-up 
display 140 is constructed according to well-known principles and the image is projected focused in front 
of vehicle such that the driver can observe the image without taking his or her eyes from the road. 
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FIGS. 13A,13B,13C illustrate one preferred method of separating an object in the blind spot from 
other objects in preparation for input into a neural network for identification. FIG. 13A illustrates a view 
of the image as seen by a side rear view camera of FIG. 1. Various filters are employed to simplify and 
idealize the view the output of which is shown in FIG. 13B. A variety of technologies exist to eliminate 
remaining background objects and isolate vehicle as shown in FIG. 13C. In one preferred method, the 
distance to the objects to the left and right of the vehicle can determined by the laser radar system 
described above. This permits elimination of objects that are not in the same plane as the blind spot 
vehicle. Any of the distance measuring schemes described above along with pattern matching or pattern 
linking techniques can be used to extract the vehicle. Other techniques involve the use of relative motion 
of the object in the blind spot that may involve the use of optical flow calculations. No one system is ideal 
unless the full three-dimensional representation of entire scene has been achieved. Therefore, a variety of 
techniques are used depending on particular problem at hand. 

FIG. 14 illustrates a lane-changing problem in congested traffic. In this illustration, the driver of 
vehicle 310 wishes to change lanes to pass vehicle 322. However, vehicle 321 is in the blind spot and if 
vehicle 310 attempts this lane change an accident may result. Using the teachings of this invention, the 
driver of vehicle 310 will be made aware either through a visual display or through warning signals, 
optical, audio and/or haptic, should the driver attempt to execute such a lane change. 

FIG. 15 is an angular perspective overhead view of a vehicle 710 about to be impacted in the side 
by an approaching vehicle 720, where vehicle 710 is equipped with a blind spot monitor or anticipatory 
sensor system showing a transmitter 730 transmitting electromagnetic, such as infrared, waves toward 
vehicle 720 This is one example of many of the use of the instant invention for exterior monitoring. The 
transmitter 730 is connected to an electronic module 740. Module 740 contains circuitry 742 to drive 
transmitter 730 and circuitry 744 to process the returned signals from receivers 734 and 736. Circuitry 744 
contains a neural computer 745, which performs the pattern recognition determination based on signals 
from receivers 734 and 736 (FIG. 15A). Receivers 734 and 736 are mounted onto the B-Pillar of the 
vehicle and are covered with a protective transparent cover. An alternate mounting location is shown as 
738 which is in the door window trim panel where the rear view mirror (not shown) is frequently attached. 
One additional advantage of this system is the ability of infrared to penetrate fog and snow better than 
visible light, which makes this technology particularly applicable for blind spot detection and anticipatory 
sensing applications. Although it is well known that infrared can be significantly attenuated by both fog 
and snow, it is less so than visual light depending on the frequency selected. (See for example L. A. Klein, 
Millimeter-Wave and Infrared Multisensor Design and S ignal Processing Artech House, Inc, Boston 1997, 
ISBN 0-89006-764-3). 

Radar systems, which may not be acceptable for use in the interior of the vehicle, are now 
commonly used in sensing applications exterior to the vehicle, police radar being one well-known 
example. Miniature radar systems are now available which are inexpensive and fit within the available 
space. Such systems are disclosed in the McEwan patents described above. One particularly advantageous 
mode of practicing the invention for these cases, therefore, is to use a CW radar or pulsed laser radar 
system, along with a CCD array. In this case the radar is used to determine distance and the CCD for 
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identification. 

In a preferred implementation, transmitter 730 is an infrared transmitter and receivers 734, 736 
and 738 are CCD transducers that receive the reflected infrared waves from vehicle 720. In the 
embodiment shown in FIG. 15, an exterior-deployed airbag 790 is shown which deploys in the event that a 
5 side impact is about to occur as described in U.S. patent application Ser. No. 09/307,883. 

In most of the applications above the assumption has been made that either a scanning spot or a 
line of light will be provided. This need not be the case. The light that is emitted to illuminate the object 
can be structured light. Structured light can take many forms starting with, for example, a rectangular or 
other macroscopic pattern of light and dark can be superimposed on the light by passing it through a filter. 
10 If a similar pattern is interposed between the reflections and the camera, a sort of pseudo-interference 
pattern can result sometimes known as Moire patterns. A similar effect can be achieved by polarizing 
transmitted light so that different parts of the object that is being illuminated are illuminated with light of 
different polarization. Once again, by viewing the reflections through a similarly polarized array, 
information can be obtained as to where the source of light came from which is illuminating a particular 
15 object. Different modulation schemes can also be used to create different patterns and the modulation can 
be varied in time for particular applications. 

As first disclosed in U.S. Pat. No. 5,653,462 for interior vehicle monitoring and U.S. patent 
application Ser. No. 08/247,760 filed May 23, 1994, for exterior monitoring, a modulated light source can 
be used to determine the distance to an object either interior or exterior of the vehicle. The basic principle 
20 is that the phase of the reflected light is compared to the phase of the transmitted light and the distance to 
the reflecting object is determined by the phase difference. There are many ways of implementing this 
principle. One that has recently been disclosed called the photonic mixing device or PMD. In this device, 
an optical filter is modulated with the same frequency and the phase that is used to modulate the 
transmitted light beam. In the PMD, this principle is executed on a pixel by pixel basis and incorporated 
25 into the CMOS array structure. Although still falling within the teachings of this invention, this results in 
an unnecessarily complicated structure. An alternate method will now be described. 

An object in the blind spot or inside a vehicle is illuminated by modulated light and reflects this 
light back to a receiver wherein the phase relations! lip between the reflected light and the transmitted light 
is a function of the distance to the reflecting surface. For every pixel, the comparison will be made to the 
30 same frequency and phase since only one source of illuminating modulated light has been used to 
illuminate the entire object. Therefore, there is no advantage in attempting to influence each pixel 
separately with the modulation frequency and phase. A similar and preferable approach is to use a single 
light valve. or electronic shutter to modulate all of the light coming back from the illuminated object. 

The technology for modulating a light valve or electronic shutter has been known for many years 
35 and is sometimes referred to as a Kerr cell or a Pockel cell. These devices are capable of being modulated 
at up to 10 billion cycles per second. For determining the distance to a vehicle in the blind spot, 
modulations between 5 and 100 MHz are needed. The higher the modulation frequency, the more accurate 
the distance to the object can be determined. However, if more than one wave length, or better one-quarter 
wavelength, exists between the host vehicle and the object., then ambiguities result. On the other hand, 
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once a longer wave length has ascertained the approximate location of the vehicle then more accurate 
determinations can be made by increasing the modulation frequency since the ambiguity will now have 
been removed. 

In one preferred embodiment of this invention, therefore, an infrared floodlight is modulated, at a 
5 frequency between 5 and 100 MHz and the returning light passes through a light valve such that amount of 
light that impinges on the CMOS array pixels is determined by a phase difference between the light valve 
and the reflected light By modulating a light valve for one frame and leaving the light valve transparent 
for a subsequent frame, the range to every point in the camera field of view can be determined based on the 
relative brightness of the corresponding pixels. 
1 0 Once the range to all of the pixels in the camera view has been determined, range-gating becomes 

a simple mathematical exercise and permits objects in the image to be easily separated for feature 
extraction processing. In this manner, many objects in the blind spot can be separated and identified 
independently. 

As mentioned above, it is frequently not possible to separate light from a broad iUumination 
15 source from sunlight, for example. It has been determined, however, that even in the presence of bright 
sunlight a reflection from a narrow beam of infrared laser fight can be observed providing a narrow notch 
frequency filter is used on the light entering the receiver. The principles described above, however, are 
still applicable since a sampling of pixels that have been significantly illuminated by the narrow laser beam 
can be observed and used for ranging. 

20 The technique of using a wide angle infrared floodlight is particularly useful at night when 

objects, especially those without self-contained lights, are difficult to observe. During bright sunlight, 
there is considerable information from the visual view taken by the cameras to perform feature extraction, 
identification, ranging etc. utilizing other techniques such as relative motion. Thus, a superior blind spot 
monitoring system will make use of different techniques depending on the environmental conditions. 

25 In more sophisticated implementations of the present invention, there can be an interaction 

between the imaging system and the aiming direction of the infrared laser beam. For example, a particular 
limited area of the image can be scanned by the infrared system when the imaging system is having 
difficulty separating one object from another. It is expected, as the various technologies described above 
evolve, that very smart blind spot anticipatory sensors and general exterior monitoring systems based on 

30 the teachings of this invention will also evolve. 

The goal is to determine the direction that a particular ray of light had when it was transmitted 
from the source. Then, by knowing which pixels were illuminated by the reflected light ray along with the 
geometry of the transducer mountings, the distance to die point of reflection off of the object can be 
determined. This requires that the light source is not collocated with the CCD array. If a particular light 
35 ray, for example, illuminates 'an object surface which is near to the source, then the reflection off of that 
surface will illuminate a pixel at a particular point on the CCD array. If the reflection of the same rav 
however occurs from a more distant surface, then a different pixel will be illuminated in the CCD array. In 
this manner, the distance from* the surface of the object to the CCD can be determined by triangulation 
formulas. Similarly, if a given pixel is illuminated in the CCD from a reflection of a particular ray of light 



35 



I 



I 



WO 01/85491 PCT/US01/14876 

r 

the turning motion of the driver. Alternately, a vibration can be induced into the steering wheel as a 
further warning to the operator not to execute a lane change. Following these activities, control is returned 
to the image acquisition and range determination activities and the process repeats. 

There has thus been shown and described a monitoring system for monitoring the exterior of the 
5 vehicle using an optical system with one or more CCD or CMOS arrays and other associated equipment 
which fulfills all the objects and advantages sought after. 

More particularly, described above is a method for determining the identification and position of 
objects exterior to a vehicle which comprises the steps of transmitting optical waves into the space 
surrounding the vehicle from one or more locations, and comparing the images of the exterior of the 

1 0 vehicle with stored images of objects external to the vehicle to determine winch of the stored images match 
most closely to the images of such objects such that the identification of the objects and their position is 
obtained based on data associated with the stored images. The optical waves may be transmitted from 
transmitter/receiver assemblies positioned at one or more locations around the exterior of the vehicle such 
that each assembly is situated where it has a good view of a particular space near the vehicle. Each 

15 assembly may comprise an optical transmitter (such as an infrared LED, an infrared LED with a diverging 
lens, a laser with a diverging lens and a scanning laser assembly, an infrared floodlight, or other light 
source) and an optical array (such as a CCD array and a CMOS array). The optical array is thus arranged 
to obtain the images of the exterior of the vehicle represented by a matrix of pixels. To enhance the 
method, prior to the comparison of the images, the output from each array can be compared with a series of 

20 stored arrays representing different objects using optical correlation techniques. Preferably, a library of 
stored images is generated by positioning an object near the vehicle, transmitting optical waves toward the 
object from one or more locations, obtaining images of the exterior of the vehicle, each from a respective 
location, associating the images with the identification and position of the object, and . repeating the 
positioning step, transmitting step, image obtaining step and associating step for the same object in 

25 different positions and for different objects in different positions. This is similar to the training and 
adaptation process described in detail in U.S. patent application Ser. No. 09/382,406 filed August 24, 1999 
on interior monitoring sy stems. 

One of the advantages of the invention is that after the identifications and positions of the objects 
are obtained, one or more systems in the vehicle may be affected based on the obtained identification and 

30 position of at least one of the objects. Such systems include a visual and/or audio warning system to alert 
the driver to the presence, position and/or velocity of objects in the blind spots as well as a system for 
adjusting the turning resistance of the steering wheel to prevent movement by the driver into the path of an 
object. Another system could be associated with the turning indicators to provide an alarm if a turning 
signal is activated when an object is present in the blind spot which would interfere with the intended turn. 

35 The image comparison may entail inputting the images or a part or form thereof into a neural 

network that provides for each image, an index of a stored image that most closely matches the inputted 
image. The index is thus utilized to locate stored information from the matched image including, inter alia, 
a locus of the center of the front of the object and an appropriate icon for display purposes. To this end, a 
display could be provided in the passenger compartment or through the use of a heads-up display to 



I 



I 



WO 01/85491 PCT/US01/14876 



from the transmitter, and the direction in which that ray of light was sent from the transmitter is known, 
then the distance to the object at the point of reflection can be determined. If each ray of light is 
individually recognizable and therefore can be correlated to the angle at which it was transmitted, then a 
full three-dimensional image can be obtained of the object that simplifies the identification problem. 
5 The coding of the light rays coming from the transmitter can be accomplished in many ways. One 

method is to polarize the light by passing the light through a filter whereby the polarization is a 
combination of the amount and angle of the polarization. This gives two dimensions that can therefore be 
used to fix the angle that the light was sent. Another method is to superimpose an analog or digital signal 
onto the light which could be done, for example, by using an addressable light valve, such as a liquid 

10 crystal filter, electrochromic filter, or, preferably, a garnet crystal array. Each pixel in this array would be 
coded such that it could be identified at the CCD. Alternately, the transmitted radiation can be AM or FM 
modulated to also provide source identification. 

The technique described above is dependent upon either changing the polarization or using the 
time or frequency domain, or a combination thereof, to identify particular transmission angles with 

15 particular reflections. Spatial patterns can also be imposed on the transmitted light that generally goes 
under the heading of structured light, as discussed above. The concept is that if a pattern is identifiable 
then either the direction of transmitted light can be determined or, if the transmission source is located on 
the same axis but axially displaced with the receiver, then the pattern expands at a different rate as it 
travels toward the object and then, by determining the size of the received pattern, the distance to the 

20 object can be determined. In some cases, Moire pattern techniques are utilized. 

A further consideration to this invention is to use the motion of the object, as determined from 
successive differential arrays, for example, to help identify that there is in fact an object in the blind spot. 
Differential motion can be used to separate various objects in the field of view and absolute motion can be 
used to eliminate the background, if desired. 

25 FIG. 16 is an exemplary flow diagram of one embodiment of this invention. The blind spot 

monitor begins by acquiring an image of the blind spot that contains an object to be identified and by 
determining the range to the object and outputs range information and image information to a feature 
extraction routine. The output from the feature extracuon routine is fed into the neural network or other 
pattern recognition algorithm. The algorithm determines the identity of object. Once the identity and 

30 range of the object is known then the display can be updated. Using current and recent information, the 
relative velocity algorithm determines the relative velocity of the object to the host vehicle. With the 
position, velocity and identity of the object in the blind spot known, an appropriate algorithm determines 
whether it is safe for a lane-changing maneuver. If the determination is yes, then control is returned to the 
image collection and ranging activities and a new image and range is determined. If the lane change 

35 determination is no, then a determination is made if the turn signal is activated (which would be indicative 
of the driver's intention to change lanes). If yes. then audio and/or visual warnings are activated. If no, 
then a determination is made if the operator has begun to change the direction of the vehicle to begin 
executing a lane change (and simply failed to activate the turn signal). If yes, then the audio and/or visual 
warnings are again activated and a haptic system begins to exert a torque on the steering wheel to oppose 
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provide a visual overview of the environment surrounding the vehicle. The icons could be general icons of 
objects in general or more specific icons indicative of the type of vehicle, etc. Moreover, the position of 
the object relative to the vehicle may be determined so that an action by the driver of the vehicle that might 
result in an accident is prevented. It is also possible to obtain information about the location of the object 

5 from the image comparison and adjust the position of one or more of the rear view mirrors based on the 
location of the object. Also, the location of the object may be obtained such that an external light source 
may be directed toward the object to permit a better identification thereof. 

In addition, the location of the locus of the center of the object exterior to the vehicle may be 
monitored by the image comparison and one or more systems in the vehicle controlled based on changes in 

1 0 the location of the locus of the center of the object exterior to the vehicle over time. This monitoring may 
entail subtracting a most recently obtained image, or a part thereof, from an immediately preceding image, 
or a corresponding part thereof, and analyzing a leading edge of changes in the images or deriving a 
correlation function which correlates the images with the object in an initial position with the most recently 
obtained images. 

15 In another method for determining the identification and position of objects external to the vehicle 

in accordance with the invention, optical waves are transmitted into a space near the vehicle from a 
plurality of locations, a plurality of images of the exterior of the vehicle are obtained, each from a 
respective location, a three-dimensional map of the exterior of the vehicle is created from the images, and a 
pattern recognition technique is applied to the map in order to determine the identification and position of 

20 the objects. The pattern recognition technique may be a neural network, fuzzy logic or an optical 
correlator or combinations thereof. The map may be obtained by utilizing a scanning laser radar system 
where the laser is operated in a pulse mode or continuous modulated mode and determining the distance 
from the object being illuminated using time-of-flight, modulated waves and phase measurements with or 
without range gating. (See for example, H. Kage, W. Freemen, Y Miyke, E. Funstsu, K. Tanaka, K. 

25 Kyuma "Artificial retina chips as on-chip image processors and gesture-oriented interfaces", Optical 
Engineering, December, 1999, Vol. 38, Number 12, ISSN 0091-3286.) 

In a method for tracking motion of a vehicle in accordance with the invention disclosed above, 
optical waves are transmitted toward the object from at least one location, a first image of a portion of the 
space exterior of the vehicle is obtained, the first image being represented by a matrix of pixels, and optical 

30 waves are transmitted toward the object from the same location(s) at a subsequent time and an additional 
image of the particular space exterior of the passenger compartment is obtained, the additional image being 
represented by a matrix of pixels. The additional image is subtracted from the first image to determine 
which pixels have changed in value. A leading edge of the changed pixels and a width of a field of the 
changed pixels is determined to thereby determine relative movement of the object from the time between 

35 which the first and additional images were taken. The first image is replaced by the additional image and 
the steps of obtaining an additional image and subtracting the additional image from the first image are 
repeated such that progressive relative motion of the object is attained. 

Also disclosed above is a method for controlling the steering system of a vehicle which comprises 
the steps of transmitting optical waves toward an object located in the vicinity of the vehicle, obtaining one 
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or more images of an exterior space proximate to the vehicle, analyzing each image to determine the 
distance between the object and the vehicle, and controlling steering system to prevent the operator from 
causing a collision with the object based on the determined distance between the object and the vehicle. 
The image may be analyzed by comparing the image of a portion of the exterior of the vehicle with stored 
5 images representing different arrangements of objects in the space proximate to the vehicle to determine 
which of the stored images match most closely to the image of the exterior of the vehicle, each stored 
image having associated data relating to the distance between the object in the image and the vehicle. The 
image comparison step may entail inputting the image or a form or part thereof into a neural network that 
provides for each such image, an index of a stored image that most closely matches the image of the 
10 exterior of the vehicle. In a particularly advantageous embodiment, the size of the object is measured and 
a vehicle system is controlled based on the determined distance between the object and the vehicle and the 
measured size of the object. 

In another method disclosed above for determining the identification and position of objects 
proximate to a vehicle, one or more images of the exterior of the space proximate to the vehicle, or part 

15 thereof, of radiation emanating from the objects proximate to the vehicle, and the images of the radiation 
emanating from the objects are compared with stored images of radiation emanating from different objects 
proximate to the vehicle to determine which of the stored images match most closely to the images of the 
exterior objects of the vehicle such that the identification of the objects and their position is obtained based 
on data associated with the stored images. In this embodiment, there is no illumination of the object with 

20 optical waves. Nevertheless, the same processes described above may be applied in conjunction with this 
method, e.g., affecting another system based on the position and identification of the objects, a library of 
stored images generated, external light source filtering, noise filtering, occupant restraint system 
deployment control and the utilization of size of the object for vehicle system control. 

Many changes, modifications, variations and other uses and applications of the subject invention 

25 will, however, become apparent to those skilled in the art after considering this specification and the 
accompanying drawings which disclose the preferred embodiments tiiereof. All such changes, 
modifications, variations and other uses and applications which do not depart from the spirit and scope of 
the invention are deemed to be covered by the invention which is limited only by the following claims. 
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CLAIMS: 

1. An arrangement for obtaining information about objects in an environment around a 
vehicle, comprising: 

light emitting means arranged on the vehicle for emitting infrared light into the environment 
5 around the vehicle; 

receiver means arranged on the vehicle for receiving infrared light from the environment around 
the vehicle; and 

measurement means coupled to said light emitting means and said receiver means for measuring 
time between emission of the infrared light by said light emitting means and reception of the infrared light 
0 by said receiver means whereby the measured time correlates to distance between the vehicle and an object 
from which the infrared light is reflected. 

2. The arrangement of claim i, wherein said light emitting means comprise an array of 
laser diodes. 

5 

3. The arrangement of claim 1, wherein said light emitting means comprise a pulsed laser. 



4. The arrangement of claim 1 ? wherein said light emitting means comprise a continuous 
laser beam directing infrared light to scan in a line and means for controlling said scanning laser beam of 
infrared light such that the infrared light traverses a volume of space near the vehicle, 

5. The arrangement of claim 4, wherein said receiver means comprise a single pixel 

receptor. 

6. The arrangement of claim 1, wherein said receiver means comprise at least one of a CCD 
array, a CMOS array and an HDRC camera, a dynamic pixel camera and an active pixel camera. 

7. The arrangement of claim 1, further comprising a processor coupled to said receiver 
means for providing an identification of the object from which light is reflected. 

8. The arrangement of claim 7, wherein said processor utilizes pattern recognition 
techniques. 

.9. The arrangement of claim 7, wherein said processor utilizes a modular neural network to 

identify the object from which light is reflected. 

10. The arrangement of claim 7, wherein said processor is arranged to create a three- 
dimensional map of a portion of the environment surrounding the vehicle. 
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1 1 . The arrangement of claim 10, wherein said processor is arranged to extract features from 
the three-dimensional map. 

* 

12. The arrangement of claim 11, further comprising a display visible to a driver of the 
5 vehicle for displaying features or representations derived from features extracted from the three- 
dimensional map. 

13. The arrangement of claim L wherein said light emitting means and said receiver means 
are collocated. 

10 

14. The arrangement of claim 1, wherein said light emitting means comprise a plurality of 
light emitting elements and said receiver means comprise at least one light receiving element. 

15. The arrangement of claim 14 ? wherein said associated light emitting elements and said at 
15 least one receiving element are spaced apart from one another. 

16. The arrangement of claim 1, wherein said receiver means comprise a notch filter for 
filtering light other than infrared light emitted by said light emitting means. 

20 17. The arrangement of claim 1, wherein said receiver means comprise a light valve. 

18. The arrangement of claim 1 , further comprising a processor coupled to said measurement 
means for determining distance between the vehicle and the object from which infrared light is reflected 
and velocity of the object based on a plurality of position measurements. 

25 

19. A system for controlling a vehicular system based on the presence of an object in an 
environment around a vehicle, comprising: 

an arrangement for obtaining information about the object, said arrangement comprising 

light emitting means arranged on the vehicle for emitting infrared light into the 
30 environment around the vehicle, and 

receiver means arranged on the vehicle for receiving infrared light from the environment 
around the vehicle; 

a vehicular system adapted to be controlled or adjusted upon the determination of the presence of 
an object in the environment around the vehicle; and 

35 a Processor coupled to said arrangement and said vehicular system for obtaining information 

about the object based at least on the infrared light received by said receiver means and controlling the 
vehicular system based on the obtained information. 

20. The system of claim 19, wherein said processor is arranged to measure time between 
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emission of the infrared light by said light emitting means and reception of the infrared light by said 
receiver means whereby the measured time correlates to distance between the vehicle and the object from 
which the infrared light is reflected. 

m 

5 21. The system of claim 19, wherein said receiver means are arranged to obtain at least one 

image of the environment around the vehicle, said processor being arranged to process said at least one 
image and identify any objects in said at least one image. 

22. The system of claim 21, wherein said processor utilizes pattern recognition techniques. 

10 

23. The system of claim 22, wherein said processor utilizes a modular neural network to 
identify any objects in said at least one image. 

24. The system of claim 19, wherein said receiver means comprise at least one of a CCD 
15 array, a CMOS array and an HDRC camera, a dynamic pixel camera and an active pixel camera. 

25. The system of claim 19, wherein said processor is arranged to create a three-dimensional 
map of a portion of the environment surrounding the vehicle based on the infrared light received by said 
receiver means. 

20 

i 

26. The system of claim 25, wherein said processor is arranged to extract features from the 
three-dimensional map. 

27. The system of claim 26, wherein said vehicular system is a display visible to a driver of 
25 the vehicle for displaying features or representations derived from features extracted from the three- 
dimensional map. 

28. The system of claim 19, wherein said vehicular system is a steering wheel having an 
adjustable turning resistance. 

30 

29. The system of claim 19. wherein said vehicular system is at least one of an audio alarm 
and a visual warning viewable by a driver of the vehicle. 
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